Clearing the htmlpurifier Cache via the Command Line

Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
Number of replies: 18
Hello!

We are running Moodle 3.1.7 (Build: 20170710). Whenever we try to clear the htmlpurifier cache through Moodle, we are met with HTTP Error 500.

To get around that, we are wondering if it is safe to remove via the command line in linux all files and subdirectories in the following path:

moodledata/cache/cachestore_file/default_application/core_htmlpurifier/? 

If so, does it need to happen in a maintenance window?

Thanks!

Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Can we take a step back... why are you trying to do this at all?

And... Error 500 tells you absolutely nothing. Enabled [[Debugging], reproduce the problem, check for additional messages and/or check your web server's error log. 
Average of ratings: -
In reply to Howard Miller

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
We are trying to do this because we are running out of disk space. The htmlpurifier cache directory is taking up the most space.
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I've never heard of this before. How much disk space are you talking about?

Anyway, as above, you need to find out what's causing that Error 500
Average of ratings: -
In reply to Howard Miller

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
The Error 500 is on the list of things need to be solved. The disk space issue is of higher priority because it is causing "error/file_not_writable" pop up and preventing us from making any changes on the site.

As a baseline the number right now on the affected file system is:

Filesystem Inodes IUsed IFree Use% Mounted on
/dev/mapper/SASGroup00-usr—local 983040 916972 66068 94% /usr/local

Also note the number of files in the htmlpurifier cache directoy

[root@epicure default_application]# pwd
/usr/local/moodledata/cache/cachestore_file/default_application
[root@epicure default_application]# ls -Ra core_htmlpurifier |wc -l
878182

The 878182 roughly correlates to the same number of inodes, so the vast majority on that file system are used in this directory.
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Sorry - I don't agree with you. The Error 500 *is* the problem and is the priority to be solved. That will tell you why your system isn't working properly.

Anyway... you seem to know what you are talking about, so I really don't understand why *at least* you haven't checked your web server's error log. 

Just seems like a very strange approach to me. If you see an obvious error - you fix it. 
Average of ratings: -
In reply to Howard Miller

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
I totally understand. Thanks for your help!
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

+1 to what Howard is suggesting!

See how much ...

cd /path/to/moodledata/

du -h ./cache

And is your cron running frequently enough?  Think recommendations now-a-days is every 1 minute.

What is setting for \core\task\cache_cleanup_task - default is every 30 min.

There are really quite a few clean up task ...

cd admin/tool/task/cli

php schedule_task.php --list |grep cleanup

to see them all and their settings as well as when last executed.

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
Unfortunately, we do run our cron that often and this is still posing a problem which is why we are looking for a different solution to solve our problem.
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

Run the task clean up.

Did it improve the situation?

If it did, could set up a cron job for the cleanup and run it more often than the main cron job.

Caches can be removed manually.  They will be rebuilt by Moodle (might slow down site from time to time). 

Guess you could setup a watch command on space/usage of the htmlpurifier Cache and have it redirect to a man made log ... adding to the log so one can see space usage over a period of time.

IF I remember correctly, 3.1.x had some issues with cron and task.

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
Unfortunately, the task clean up didn't help. But it does makes sense that there could be some problems with cron and task because none of the traditional routes are working. It's good to know that the cache can be removed manually and will be rebuilt by Moodle, which is what we need to do at the moment. Thanks for your help!
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

Questions asked about hosting and how + if provider offered attached devices/drives wasn't to be nosey - but a work-around solution to issue.

Some hosting providers do have inode limits for customers with lower end hosting options.  If you have 'cheapo' with smaller inode caps/limits, providers won't do anything to help the issue.   Solution ... for that provider ... upgrade hosting setup to a system that doesn't have the lower inode caps.

Reason I asked about if provider offered attached devices/drives ... it is possible to break apart what moodle does where.   Most of moodledata needs to be fast and best place for it is in close proximity to the code.  Moodle has settings for caching options/choices ... memcached, etc.  A 'do it yourself' kinda thing could involve moodledata/cache directory really residing on an attached drive/device thus not under servers inode restrictions.

If possible ... something for you to add to your list! smile

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Clearing the htmlpurifier Cache via the Command Line

by Cana Owens -
We don't have a hosting provider. We do everything in house. Just as you guessed, we are reaching our inode limit. Looking into what Moodle does where is a great suggestion and will added to the list! smile
Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

Have you looked at one of those files (it's contents) in that cache?

Out of curiosity I checked a site ... not really a large busy site ... one for a corp that uses Moodle quizzes.  Cache files contained references to processing a quiz attempt ... in stages.  There were probably others for diff mods etc. but on that site no error 500's ... no inode limits hit.

Just a guess, but ... is this 'finals week' for your entity? or something going on in your site that would result in a bump up of such caching?

Food for thought/investigation? smile

'SoS', Ken

Average of ratings: -
In reply to Cana Owens

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

How are you hosted?

And does hosting provider offer the attachment of data disk?

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Clearing the htmlpurifier Cache via the Command Line

by Daniel G. Gonzalez -

I have arrived to this forum because of a similar problem. My 'inode' usage seems to be full (101% of its 200.000 limit)

I have a website with loads of quizzes (dozens) and 1.400 users. Which seems to match what Ken Task mentioned...

Now seems that website stopped working because of that: limit reached.

I have no idea how to start solving this. I can acces files through my 'Files Manager' but what must I delete?

(Please note my students don't need to keep their quizz attemps for ever. It's just quizzes for preparation, not quizzes for assessment, so I could delete that usage history if needed)

Please, help..

Average of ratings: -
In reply to Daniel G. Gonzalez

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

@Daniel ... no need to ask or expect response via PM in this moodle ... response here for all to see ... others could have better suggestions ... below is my 2 cents!

Caution you about 'piggy backing' on someone else's post/problem.   Might seem the same but unless your server setup is on same hosting provider or setup exactly as the other poster in this thread, you could be going off and down a rabbit hole that won't lead to resolution of your problem.

Please do as Howard has suggested in this thread ... if you are getting error 500's, find out why.

Files in something dynamic (changes constantly) like Moodle isn't easy.   Not only the contents of moodledata/filedir/ but the database (mdl_files table) is involved and through queries of DB is how Moodle finds files.

To solve your issue think I'd do a query of DB first to find all moodle course backups in moodledata/filedir/

Something like the following ... you would have to adjust accordingly for your setup/server:

mysql -u root -p -e "use moodle;select id,component,userid,filename,filesize,timemodified from mdl_files where filename like '%.mbz';" > allbackups.txt;cat allbackups.txt

The text file it outputs to can be used to find backups that are dups and large.

Example from real in production site:

id    component    userid    filename    filesize    timemodified

162051    backup    2    backup-moodle2-course-21-car002-20130716-2311-nu.mbz    25754656    1374030713
163827    backup    2    backup-moodle2-course-21-car002-20130717-2311-nu.mbz    25754658    1374117115
163866    backup    2    backup-moodle2-course-21-car002-20130718-2311-nu.mbz    25754656    1374203510
164170    backup    2    backup-moodle2-course-21-car002-20130719-2311-nu.mbz    25754656    1374289911
164210    backup    2    backup-moodle2-course-21-car002-20130720-2311-nu.mbz    25754656    1374376309

You can see by example provided this one course has several old backups and are a little over 25+Megs in size.

Finding them and removing them from admin interface is difficult ... admin level user doesn't see teachers private files unless you login as them.

As per example: Logging into server as that teacher and going to course ID 21 which has a shortcourse name of car002 you will be able to see those .mbz files and delete them.

But, that doesn't automatically free up that space ... those backup mbz files are moved to moodledata/trashdir where they remain for 4 days I think, then task in cron job to empty trash kicks in to finally deletes them.

OR ... since you need space now for inode limits, one can manually remove all contents of moodledata/trashdir/

The database will catch up with cron jobs and task.

Manually removing should never be first resort ... but sounds like you are in a crunch and have no choice at present.

When you get this resolved, I'd look into finding more space on an attached device/drive/mountpoint which has inode limits of it's own ... hopefully larger than what you have now.

Above depends upon how hosted, with whom, does hoster provide such things, etc.   All of that known only to you at present.

Good luck!

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Clearing the htmlpurifier Cache via the Command Line

by Daniel G. Gonzalez -
Great, Ken. THANKS a lot for quick and detailed reply.

(And Sorry! for  my previous PM I sent due to my lack of experience in this forums. I thought it was public). Also sorry for my English as non-native speaker.

Let me share my experience as I finally SOLVED the problem, and probably it can be similar maybe to other's arriving to this same forum after googling about the 'inodes'.

First I will tell my problem, then I will put in context for other users who can be using also 'quizzes' on their moodles and finally the (very) simple solution.

I'm not a developer. I'm a teacher at Uni and I just use Moodle for my work, so I decided it was fun to create and run a home-made website.

Currently, after two years, this website hosts one single course with 1.400 users.

My problem was that suddenly I ran out of inodes in my hosting company ('Hostinger', by the way). I reached 101% of my 200.000 inodes permitted.

Never heard about those 'inodes' in my whole life. When you use Moodle at Uni you never think about all those other limits

Inodes seem to be the meta data every single file or folder needs to be hosted in your website. So, no matter how small they are, as long as they exist, they use one. So, for a website like mine (quite small, with no more than 900 MB used, so, less than I GB) with a Premium service plan I payed it seemed to be quite surprising.

So, where did all those inodes come from?

The answer is the structure and usage of my website, which is full of Quizzes (dozens of them, with 'randomly selected questions' from database) and 1.400 active users. This seems to create a lot of tiny 'cache' files every day. The reason is related to the kind of quizzes used, as they are created 'randomly' for every single user, the amount of information stored tend to grow very fast.

The solution seemed to be just clearing those caches, which is something I discover it's quite easy from inside the Moodle itself, but the problem is that when your inodes are full, you cannot even enter your website! (No PhP can run, so no access is allowed for any user, included administrators). So I was out of my own website and couldn't enter as Admin. Just surf the directories using the File Manager...

So: what was the solution in my case?

I used File Manager to delete another folder (out of Moodle, just an unused folder with rubbish, it was a secondary moodle folder I installed once in the past but never used again). This folder was stored in the same hosting plan and that helped reducing my inodes below the 100% limit.

As long as you get below the limit, you can finally enter as Admin. And when you're in, go to:

SITE ADMINISTRATION / DEVELOPMENT / PURGE ALL CACHES

something like this:

https://www.MYWEBSITE.com/admin/purgecaches.php

And purge all of of them:

This drastically reduced the amount of inodes used to just 14% of my 200.000 limit.

Great! very happy...

BUT:

About the obvious conclusion ("...always purge the caches of your Moodle site before reaching the 100% limit of your storage plan...') A second conclusion I got about this is as follows:

Always keep also folder (unused) with rubbish or files you don't need –In my case, that secondary and unused Moodle site– which you can delete in order to create extra space in case of an Emergency.

I know other hosting companies do alert their customers when inodes limit is about to be reached (at 90 or 95%) but that was not my case. So that's why I think having the secondary and unused moodle can be a simple lifeguard!

Thanks to all users in the forum. Your help is very valuable!
Average of ratings: Useful (1)
In reply to Daniel G. Gonzalez

Re: Clearing the htmlpurifier Cache via the Command Line

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for your sharing ... and solution.  Yep.  Clearing caches BTW is a task/cron job.  Problem however when one cannot get to moodle admin via browser.

@Daniel

Tip for others who do have ssh/command line access ... in moodlecode/admin/cli/ there is a purge_caches.php script ... from moodlecode/admin/cli/ execute like:

php purge_caches.php [ENTER]

All the scripts in CLI are command line only ... don't involve your web service ... just php-cli and the DB.

Tip for anyone looking at hosting features .... make sure whatever one signs up for, ssh access is possible.  Even if you don't use cli, someone helping you could.  It ain't all that scary anyway! smile

'SoS', Ken




Average of ratings: Useful (1)