Moodle in English: One node seems to behave worse than others.

One node seems to behave worse than others.

by Ángel Orosa - Monday, September 2, 2019, 8:10 PM

Hello,

We are experiencing an issue that repeats sometimes but it affects to the end users, I hope you can help us.

In our company, we have the following configuration:

Four Moodle 3.5.4 nodes with PHP 7.1 and Apache 2.4.
One MariaDB 5.5.52 server.
Operating system is Red Hat 7.3.
Two Moodles
One balancer that manage connections through the servers.
Cron is run each five minutes among all nodes (I mean, first node runs at 00, 20, 40 each hour, second node at 05, 25, 45 each hour, and so on...)

Sometimes it happens that one node start to perform worse than the others, with the following symptoms:

Sessions in MariaDB from this node slowly start to increase: they usually start appearing as "Sleep" and 0 seconds in this state but with time the start to increase the time Sleeping and also the amount of sessions.
In the PHP node, http sessions start to increase also and consecuently, the load of the server.

Once this behaviour is noticed, performance through this node is worse than others, response time and even timeouts.

There is not apparent reason for this behaviour, these servers are dedicated to Moodle, there is not any error in /var/log/messages or any other log. Also, is not always the same node.

Has this happened to anyone before?

Thanks in advance.

Ángel O.

Re: One node seems to behave worse than others.

by Howard Miller - Tuesday, September 3, 2019, 3:36 AM

This is interesting.... we've been seeing something broadly similar. Lots of questions...

* What are you using to share files? NFS? Which version? What's the configuration?
* What are using for cacheing? What are your MUC settings?
* What sort of load balancer are you using?
* Do you know if you're seeing load average rise on the web servers?
* When you say "sessions" are we all talking about the same thing... do you really mean database "connections" or do you really mean "sessions" (i.e. you have configured Moodle's session handling to use the database)?
* How often does this happen?
* What do you do to "fix" it? Even if a temporary solution?

Average of ratings: -

Re: One node seems to behave worse than others.

by Ángel Orosa - Tuesday, September 3, 2019, 9:00 PM

Hello, Howard,

Thanks for your answer, you'll find the answer to the questions below:

What are you using to share files? NFS? Which version? What's the configuration?

Moodle Data is shared between nodes mounting a CIFS unit shared from a Windows Server.
CIFS package installed is cifs-utils-6.2-9.el7.x86_64.
The configuration is an entry in the fstab file: //UNIT/moodle /mnt/moodle_data cifs _netdev,defaults,uid=moodle,gid=moodle,credentials=/etc/.moodle,domain=company.local 0 0

What are using for cacheing? What are your MUC settings?

We don't have a common cache as far as I know.

What sort of load balancer are you using?

The balancer that distributes the requests is a physical A10.

Do you know if you're seeing load average rise on the web servers?

As soon as the amount of httpd processes increases so does the load in the server.

When you say "sessions" are we all talking about the same thing... do you really mean database "connections" or do you really mean "sessions" (i.e. you have configured Moodle's session handling to use the database)?

You are right, I meant database connections.

How often does this happen?

It does not have a pattern, actually, it can be weeks without any incident.
At the begining I though it could be caused by Ghostscript because sometimes there was gs processes started many days before but it does not seem this the issue.

What do you do to "fix" it? Even if a temporary solution?

The temporary solution is take out the server from the balancer and restart it.

If it is not restarted sometimes the amount of httpd processes is reduced but as soon as it is added again in the balance it starts to behave badly again.

Thanks again.

Average of ratings: -

Re: One node seems to behave worse than others.

by Howard Miller - Tuesday, September 3, 2019, 9:06 PM

The first thing you need to do is to get a proper Cache. Redis is as good as any.

Currently your application cache is saving to your CIFS share. This isn't good. It won't be fast be enough. I would resolve that before you do anything else. Other things to do to reduce the load on the shared drive is to move your lock factory to the database and to move sessions (real sessions) to Redis. Don't be tempted to use database sessions - it doesn't work.

Average of ratings: -

Re: One node seems to behave worse than others.

by Ángel Orosa - Wednesday, September 4, 2019, 4:56 PM

Hello, Howard,

Thank you very much for your recommendations. I'll look into them.

Average of ratings: -

Hardware and performance

One node seems to behave worse than others.