"Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

"Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Mich Ere -
Number of replies: 10
We've been getting the following alerts every hour, since upgrading to 4.2.  Never got them with 4.1.
We're all up to date with our VPS Server requirements, including MariaDB 10.6 and PHP 8.0.28

Are the following long running processes normal?  Should we just up the threshold for the Alerts, as our VPS Host and cPanel have suggested?  Or do these long-running processes need to be fixed somehow?  Our VPS host killed them once, and they simply restarted again.  We just want to make sure we're not simply hiding/ignoring a persistent process that's not correct.

"lfd on server.[name].org: Excessive resource usage: [name] (98819 (Parent PID:98819))"
Account:  [name]
Resource: Process Time
Exceeded: 89517 > 1800 (seconds)
Executable: /usr/lib/systemd/systemd
Command Line: /usr/lib/systemd/systemd --user
PID: 98819 (Parent PID:98819)
Killed: No

and

"lfd on server.[name].org: Excessive resource usage: [name] (98820 (Parent PID:98819))"
Account: [name]
Resource: Process Time
Exceeded: 89517 > 1800 (seconds)
Executable: /usr/lib/systemd/systemd
Command Line: (sd-pam)
PID: 98820 (Parent PID:98819)
Killed: No

The above processes are also listed at the top of the following Alert,

"lfd on server.[name],org:  Excessive processes running under user [name]"

which lists twelve (12) additional processes, though the others indicate that they've been running less than the default "Keep alive" setting of 3-minutes listed under "Admin > Server > Task processing."

FYI, I'm not an expert Admin, so please take this into consideration if making any ssh command suggestions.

Thanks in advance,
Mich
Average of ratings: -
In reply to Mich Ere

Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Are you sure that you are on a fully privileged Unix environment as the true super-user 'root' and not in some restricted root shell (chroot) or some simulation inside a higher OS? What is the Unix flavour? If Linux, which distribution? I can't remember systemd communicating with me directly.

What are the processes systemd complaining of? If they are simply PIDs you must get the corresponding commands.

I did a web search and came across this: https://forums.cpanel.net/threads/excessive-resource-usage-executable-usr-lib-systemd-systemd-command-line-sd-pam.697913/. I have absolutely no experience on Cpanel, in case this is a Cpanel thing.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Mich Ere -
Thanks for the feedback and link, much appreciated.

We're running AlmaLinux 8.7 and PHP 8.0.28 with Normal shell enabled, not jailed.

cPanel just informed my Host that the systemd and sd-pam are:

"Both processes are generally benign and log running sessions of either is entirely within the scope of expectations.
systemd should always be running... it is the system through which all other services within the server are run.
systemd is a system and service manager for Linux operating systems. When run as first process on boot (as PID 1), it acts as init system that brings up and maintains userspace
services.
SD-PAM is a subprocess of systemd that managed services and processes that require pam authentication."

Still, what's disturbing is the fact that we weren't getting harassed by these lfd "Excessive resource usage" warnings every hour before we upgraded to Moodle 4.2. So the upgrade somehow triggered these two warnings, plus the third, "Excessive processes running" warning.

Thus, is there something from Moodle's end that we can or need to do to stop these? My VPS Host is checking into whitelisting these alerts and upping the number of concurrent processes, but not sure if a better fix is required, since it wasn't happening before.

If nothing else, I assume this is something that Moodle needs to be aware of, at the very least, since it was triggered by the upgrade.

Thanks again சிரி
Mich
Average of ratings: -
In reply to Mich Ere

Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Somebody who knows both sides, your server environment and Moodle, need to look in first-hand. Alma Linux is not that exotic, if it stays close to RedHat, but the Systemd setup is. And then, Moodle 4.2 is only a couple of days old - you are clearly on the bleeding edge.
மின்னக்கூடிய

No, sorry, I have no idea.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Mich Ere -
Appreciate the effort!
Regards,
Mich
Average of ratings: -
In reply to Mich Ere

Ri: Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
there is a new setting in 4.2 ((https://tracker.moodle.org/browse/MDL-77186)

Site administration > Server > Tasks > Task processing > Keep alive time

The default is set to 3minutes and it’s the main cause of the cron log constantly (each second) filling of messages after the tasks run once. I modified it to 0 and the cron behaviour it’s back to the “old way”.

As a side effect of setting this to zero, my server uptime is lowering … could be this to leverage also your excessive processes running ?

Average of ratings: Useful (1)
In reply to Sergio Rabellino

Re: Ri: Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Mich Ere -
Yes, setting it to "0" worked!  Thanks, Sergio!  The hourly warnings have stopped for a couple of hours now.  Our Moodle has also disappeared from the results of running the commands:  ps auxf | grep "sd-pam" and "ps auxf | grep systemd" that were recommended in the link that Visvanath referred me to.

The environmental impact of having the server running constantly was also a concern, so this is great!

My only worry is if turning this off is now impacting other tasks that we need to know about.  Sadly, I don't know enough about Moodle or 4.2 to understand if we only "use dedicated task runners" or not, as mentioned in the Setting's description.  Our Moodle is small with just a couple courses using the Activities of Page, Lesson, Quiz, Database, Feedback, and Moodle's media player, plus some "Additional plugins" with at least 4.0 compatibility that all seem to be working fine, so I imagine it's using "dedicated task runners", but I can't say for sure.

Thanks again சிரி
Average of ratings: -
In reply to Mich Ere

Ri: Re: Ri: Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
If I understood correctly that param, having it set to 1 minute and having the cron calling every ome minute, you are sure that a task will be executed promptly (e.g. completions). A cron task will stay for keep alive time and immediately after it’s end a new one will start.
If you can wait a minute (in linux there will be a. cron with * * * * *) then you’re ok, i think, with a setup to zero. Elsewhere if you have enough computing resources or won’t wait for a task to be done, then let the default untouched.
Average of ratings: Useful (1)
In reply to Sergio Rabellino

Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Sergio

This is bleeding edge! I don't understand the details, but am I correct in saying that this "patch" increases the time resolution of Moodle from 1 min to 1 sec, but increase the CPU load?

I've started a discussion in the H&P forum on this. See MDL-77189 "Add a keep-alive setting to admin/cli/cron.php", Release 4.2. Effect on performance?
Average of ratings: -
In reply to Visvanath Ratnaweera

Ri: Re: "Excessive Resource Usage" and "Excessive Processes Running" LFD Alerts since Upgrade to 4.2

Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
I own a single site on M4.2 and after updating to from 4.1.4 to 4.2 I saw a constant uptime of 0.30/0.40, even without user activities on the moodle hosted on this server, which usually scores 0.02/0.08 ...
Now that i set up the cron keepalive time from 3 minutes to zero, the uptime has returned to "normal" values.

I'm managing this 4.2 instance for experiencing the new features (access is restricted to Italian Moodle User Association members), so I'm not so concerned about performances and I didn't realize it until I read this thread.
You are right in your description of the problem: probably in low-performance hosting this parameter has to be managed to balance between responsiveness and cpu load.

Let us see if anyone on HQ has some hints about it.
Average of ratings: -