Installation Size Question

Installation Size Question

by William Stewart -
Number of replies: 8

Just wanted to get some broader perspective on the relative sizes of people's Moodle installations. Yes, tons of variability, I know, but I am curious! 

I've been on a mission in reducing our installation's footprint over the past year and made significant progress, though I'm fairly certain if I re-created the installation from scratch and rebuilt everything, I could get it down to be pretty small (i.e. 2-3Gb total).

Our installation.

Moodle 3.2.1+/php7

Usually supports 120-150 users per 8 week session.

40 courses, the largest size component of each is mp3 files (yes, I've scaled them ALL down in quality).

We don't archive courses. Export records and reset courses every quarter. Don't have course back ups turned on, rather do daily server backups that are downloaded. We are a language program with many recording assignments so the site swells by about 1Gb, maybe 1.5Gb every session due to mp3 submissions from students which is why we don't really archive courses. 


Size is currently :

Roughly 1Gb SQL export (log store is 90% of that-been considering truncating it after keeping a copy but have never pulled that trigger yet)

6Gb data directory 


The installation has been running for 8 years. When I took it over (year 7), the site was at 20Gb and now around 7Gb. (That's right, I've reduced it by 13Gb!). This week I deleted thousands of files in the legacy course files folders which doesn't seem to have made much of an impact (or it has really been pushed out of the temp or trash dir yet).

Anyway, looking forward to hearing about the relative storage space other installations have.

Average of ratings: -
In reply to William Stewart

Re: Installation Size Question

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

How big a disk does a day of your time buy? wink

I'm being slightly facetious - getting rid of unused course / content / users et al is probably sensible but being small isn't an end in itself. Disk space is incredibly cheap. 

Average of ratings: -
In reply to Howard Miller

Re: Installation Size Question

by William Stewart -

Howard,

There's some back story to the disk space. When I first started, the backups were 40Gb and I had no idea why they were so large given that the amount of users and average course content was relatively small! The previous admin had the server running at about 90% capacity which would then easily go over and down the site temporarily with 3-4 back ups plus the actual site. Our host was also equally curious as to why the backups were so large! More problematically, backups weren't even done daily, rather every other day.

That was the impetus for the long journey in sleuthing and reducing size.  So 40Gb down to 7Gb has been a pretty big deal, though the 40Gb was due to there being 2 backup processes running back then-one from Softaculous, and then the regular server backup which would result in 40Gb. No logs or documentation from the previous admin when I took over of anything he did for 7 years sad.

My director tasked me with going down the rabbit hole to simply work backwards and try to get the size down to avoid any future problems with the site going down-and it's led me the current 7. I'm getting to the end of that journey, but I'm simply trying to do what I can within reason. Rebuilding the site is beyond the time I have, and not worth the effort given the disk space we have wink. But it makes it much easier to download backups (my bandwidth at work isn't great and probably had limits imposed by the network admins-it's a government bureaucracy to try and get it changed. Their answer is to blame our server rather than admit it is their doing! I get way better speeds out of the office!

Average of ratings: -
In reply to William Stewart

Re: Installation Size Question

by Ken Task -
Picture of Particularly helpful Moodlers

First, think you are asking a lot of work of folks to provide accurate data.   But ... understand why you are asking ....

The fact you mentioned 'legacy' means the site was migrated from 1.9 to a 2 at some point.   That first step in migration did massive changes to data directory and how Moodle links to files.   Remember, back then if you cloned a course (backup) and restored that course to begin a new 8 week session, that course was legacy.

But, it sounds like you don't do the 'backup then restore to new course', but rather develop new courses ... which means no legacy now.  It's turned off and you've removed from moodledata directory legacy types - or at least via the Moodle Admin UI.

Check out your moodledata directory ... are there still old course ID numbers in there?

What could get kinda stuck are 1.9 backups ...  in filedir and data of DB.

A db query of mdl_files looking for backup*.zip might find something there.

This:

We don't archive courses. Export records and reset courses every quarter

If you don't reset a course the students work is still associated with the course.   If they uploaded files  as part of an assignment those files are still linked ... you don't archive courses .... thus their files might still be hangin' round.   Yet, student is long gone ... over 7 years would hope they have gotten whatever certification and have moved on. ;)   How many hidden courses are in your site?

Not sure there are two moodle sites 'exactly' the same unless cloned ... how it's used is typical to you ... thus size stuff/space usage etc.  unique to your site.

Think, in order for you to find information that would be useful, whomever shared information about their space usage they would have to run their site pretty much the same as you.   Now what are the chances of that? ;)

Howard is right ... disk/space is cheap today. ;)  If you are doing all you can do to keep the site 'lean an mean' ... why sweat the small stuff? ;)

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Installation Size Question

by William Stewart -

Ken,

Actually most of the courses still retain the legacy folder associated with them. I'm considering rebuilding these courses in a new shell to get rid of them little by little. It isn't difficult or that time consuming to do, but, I just have to make time to do them. Dates on virtually all of the legacy course files are from 2013-and none of that material is used any more. I just wish Moodle had a way to delete folders in the GUI versus only individual files. It took a few days to get done.

I know that no two sites are the same, I was more curious if people are concerned with keeping them lean, or otherwise don't mind if they grow in size. This is probably the first time spring cleaning has occurred with the site in 8 years.

I'm just trying to clean out and reduce as much as I can, and then pretty much leave it alone; I'm very much a minimalist smile.

Average of ratings: -
In reply to William Stewart

Re: Installation Size Question

by Ken Task -
Picture of Particularly helpful Moodlers

1.9 backups?

select contenthash,filename,filesize from `mdl_files` where (`component` like "backup" and `filename` like "%.zip%")

Legacy files ... this might give you some incentive to clean up/remove old courses dated 2013, etc.

select contenthash,filename,filesize from `mdl_files` where `filearea` like "legacy"

"Dates on virtually all of the legacy course files are from 2013-and none of that material is used any more"

Why keep them then?   Make a backup for archival purposes ... download ... remove.

"people are concerned with keeping them lean"

I'd say so, yes ... but not overly so.  Kinda depends upon the Moodle Admin.

The issue of space and how lean and mean is really only your concern ... you know that keeping a site 'lean and mean' usually equates to better performance, etc., but teachers and students may not see it that way nor be concerned as long as they don't see errors and they can do what needs to be done without 'roadblocks' and un-reasonable waiting.

Afraid 'perfection' isn't quite possible. ;)

Can you install moosh ... command line swiss-army knife for Moode.

It has several commands that deal with files, file system, orphaned files, etc.. and can help sort stuff out.

http://moosh-online.com/

http://moosh-online.com/commands/

There's also adhoc DB tool and queries.   With a query to show the largest tables you might decide to reduce the number of months, etc. for stats/logs to be kept.   Did that on a slow site and discovered records from years ago ... deleted records below a epoch time stamp and reduced DB dump by  1.3 gig. = leaner DB = better performance?

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Installation Size Question

by William Stewart -

Ken,

As always, thanks for sharing! I'll take a look and see what I can do. Today I did the conceptually easier route of simply rebuilding some of the course shells and deleting the old ones that contained the legacy course file folders. I have some rough estimates of how much it may have cleaned out but I'll see what the output is in a couple of days. I'm almost at the end of this particular effort so thanks for all the advice!

Average of ratings: -
In reply to William Stewart

Re: Installation Size Question

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

William, I run my own moodle.  I have about 150 students every 16-week period.  I have around 8-10 courses.

My moodle.sql file size is about 1/2 gigabyte, which was new 15 months ago.  I leave logging turned on.

My moodledata folder is about 1 gigabyte.  This includes a good share of student uploads of Excel, Word, and miscellaneous files.  However, I do not include most of my own course content in moodle, I manage my webpages outside of moodle and link to these from within moodle.  Having said this, about all of my course content external to moodle is around a gigabyte.  But here is the catch, all of my videos (mp4's), maybe 300 or more, are on a university video server.  So these are not included in my moodledata folder and not in my backups.  My quick estimate is that I have around 40GB of files.  As I said, these are on a video server, but also on my Mac which I backup regularly.

My previous moodle installation, which spanned the entire moodle 2.0 versions, ended up having a moodledata folder of 3.5GB, and a sql database of around 3GB.  Same situations about my own content and videos.

Average of ratings: -
In reply to Rick Jerz

Re: Installation Size Question

by William Stewart -

Rick,


Thanks for sharing some numbers for perspective. 

 Some simple and totally inaccurate guesstimates makes me think our total installation is about 4-5.5Gb in size. Just a little bit more to see if I can reach that ballpark wink. 80% reduction in size isn't too shabby.

Average of ratings: -