I am looking at a powerful machine which became unusable recently. Not my machine. It had the bad luck of happy system administrators, who were happily doing other things. I am just investigating after disaster struck. So, need to start with vague information.
First thing I found was that moodledata/lock contains 500 k files! Apparently the admins had to quadruple Kern.maxvnodes to some 30 million in October. Going through various old log files I found the same ominous message **** File has unbalanced q/Q operators (too many Q's) **** in millions. Web searches on this string point to Ghostscript, for example https://bugs.ghostscript.com/show_bug.cgi?id=694310.
So the first question is, could the two things be related?
Interestingly almost all the lock files have date stamps from June and July this year. Going back in time I came to know that they've activated assignfeedback_editpdf early this year and installed Ghostscript somewhere in April. And they conduct assignments mass-scale where students upload PDFs. That is where my suspicion comes from.
Since the lock files are months old, I deleted the lock files and recovered a million vnodes! (Yes, stopped the cron job before doing that.)
So the second question is, how many lock files are "normal" in a busy site where hundreds of students upload PDFs?
The cron job runs every 2 min. The OS is FreeBSD 13. Web server Apache 2.4, DB server MySQL 5.7, PHP 7.3, configured mpm. All the web server instances are in a single Jail, to which I have super user access.
Related discussions in these forums: