Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Mireille Raad -
Number of replies: 14
Hello guys,

I am trying to troubleshoot the performance of a moodle demo install .
I am still running moodle on a test server with only a bunch of people with access on it (max 20 )

Usually it is me and like 3 people viewing the site.

The weird thing that is happening is that in the debug info at the bottom of the page i am getting the following :

30.490213 secs
RAM: 24.2MB
Included 69 files
ticks: 3049
user: 357
sys: 33
cuser: 0
csys: 0
Load average: 0.92
Record cache hit/miss ratio : 0/0

on the same page on other cases, i am getting :
0.681407 secs
RAM: 18.3MB
Included 62 files
ticks: 69
user: 52
sys: 3
cuser: 0
csys: 0
Load average: 0.33
Record cache hit/miss ratio : 0/0

can someone please help me what the above info means ? sys, csys, cuser, ticks ?
is that normal ? what could generate such a number of users ? any settings i should create ? (flushing users or something )

thanks for your help in advance, i appreciate your help




Average of ratings: -
In reply to Mireille Raad

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Moodle is using the PHP function posix_times http://uk2.php.net/manual/en/function.posix-times.php. Sadly the documentation for that is not very helpful.

http://en.wikipedia.org/wiki/CPU_time, and some of the pages it links to, may help to explain.

I don't know why you are seeing such wide variation in values.
Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Mireille Raad -
Thanks Tim for your reply.

Checking now the links.

and yes, regarding the wide variation , it is really puzzling .
When i pinpoint it , i will make sure to post about it for further reference.

Thanks again for your help
Average of ratings: -
In reply to Mireille Raad

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hi,

Sorry for replying in an old thread. I am a moodle new bie and have never worked in PHP either.

I am right now deploying a moodle instance for a client in their intranet.

The deployment is using multiple server(web and database). The instance is working fine and mysql clustering and master master replication is done using percona.

The only issue I am facing is the application is very slow. This being on intranet is even strange.

I have followed all guides written and turned off designer mode and everything. 

I displayed the logs in the footer,and I can see that the ticks is very high and load average is about 0.6-0.7

Apologies I cannot paste the whole output as this forum is blocked in their firewall.

What I wanted to understand is why is the number of ticks so high? Is it a processor issue ?


All servers have 4GB RAM with 2vCPS. They have a windows server and have provided us with linux (ubuntu 12.04) based VM(vmWare).

All the servers are on same network(x.x.x.121,x.x.x.122,...)

I ran the sysbench for processor performance and for 20000 prime numbers, it took 26.8secs.


Any pointers would be greatly helpful.


Thanks.

Average of ratings: -
In reply to rushabh hathi

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

This message gets the data using the PHP function posix_times. See get_performance_info in lib/moodlelib.php.

Sadly the documentation at http://php.net/manual/en/function.posix-times.php assumes you already know what the numbers mean.

These are standard performance numbers reported by Linux/Unix systems. I was trying to Google for a good explanation, but did not find one quickly. I hope you have better luck.

Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hi Tim,

Thanks for the response. Unfortunately I am no able to figure out why the page load time is very high.

Even when I click on site administration. , the Ajax request takes about 12 secs .

Can you point me in right direction ?

My guess is server is taking too long to generate HTML as when I profile using firebug , highest time is taken by php files as it keeps on waiting .

How do I debug ?


Thanks again and apologies for vague questions !

Average of ratings: -
In reply to rushabh hathi

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Michael Penney -

Are you finding it slow when logged in as a student?

On a clustered system there is a certain amount of latency due to the communication between the app server and the db server. Also, there are a lot of things the system does when an admin logs in that it doesn't do when a student or teacher logs in.

I like using a monitoring tool* that tells me what average app server response time and average end user page load times are, because of this. 

For example, on our cluster with 300 users on right now (low period between semesters) we have app server response times around 400ms, and that can go to ~600ms with 10k users on. Point is there is a certain latency in just getting the data back and forth between servers. We only do Master-Slave, not Master-Master, with Master-Master at some point drive i/o may slow you down more (are you running SSD drives?)

I have a question about your environment - you mention that you have Ubuntu and Windows running in the mix? Are you running Ubuntu on a Windows server via Visualization? This is where a monitoring tool that gives you detailed reports of where your latency is coming from is very useful. If you are running virtual app servers on the same physical server, there are some reasons to do this for resiliency and scalability, but it will have some performance implications especially when you are testing it with a small set of users.

*Mainly, we use NewRelic on our system (26k+ users, during peak times 5k-10k concurrent) for 'live' performance reporting and some other tools for forensics.

Average of ratings: -
In reply to Michael Penney

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hi Micheal,

I am not sure about times when logged in as student. We didnt have data as it was a fresh install. I will check and make sure I update the thread.

As far as latency goes, I agree but a load time of about 5 secs for the landing page of the admin is according to me too high and indicates some other issue. In the performance logs, I checked the db query time and its in the order of 0.02secs which leads me to believe there is no issue there(please do correct me If I am mistaken here).

Also, the fastest page load is the home page for admin which as I mentioned is about 5secs. 

As far as environment is concerned, yes the client have windows server and we are given Ubuntu using vmWare.

I am not sure If i would be allowed to use new relic. If you can point me to some other monitoring tool,I can request the client to do so but even there I am not sure they will agree.

I will anyway check new relic and any informatio it gives should be useful.

Just to add I am using php fast cgi. I am also using LDAP authentication.Can these be causing any issues ?

Average of ratings: -
In reply to rushabh hathi

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Michael Penney -

Hi Rushabh, I suggest created a student account and check load times there. Building/rendering the administration menu takes some time, students/teachers/regular users it's much faster.

There are other monitoring tools of course, I suggest you have your decision maker look at a demo of Newrelic and they will probably agree they need something like it -  a decent graphical real-time monitoring tool removes a lot of the mystery about performance. We also use xdebug, but only forensically (generally for looking at specific user and/or script issues) as it's very detailed, but getting those details come with their own load issues.

Generally, when I've seen high app server load times it's been related to disk i/o, but different environments generally have different limits.

IMO, running a production site without a real time monitoring tool is like driving a car with a broken speedometer - you're always guessing without actual data.

Overall, I think you also need a monitoring tool to root out the source of your latency because not many people are going to be running Moodle the way you are (a Ubuntu VM on a Windows Server)  and depending on the details of that environment latency could be from a wide variety of different things most admins will not see. I mention this because we're running Moodle on SmartOS right now, and often Linux based advice is only partly applicable.

If I was only given a Windows server to run Moodle on, I'd run it directly on IIS, not on Linux in a VM because I think there would be latency inherent in running Linux on top of Windows, but I could be wrong, need data to know smile.

Average of ratings: -
In reply to rushabh hathi

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I just noticed that you have not yet told us which version of Moodle you are using. It matters!

Nagios is a common free monitoring tool

I agree with Michael that running an Ubuntu VM on a Windows server does not seem like a  good idea.

Probably the thing that is killing you is disc IO.

In particular, how are you hosting Moodledata?

Also, what cache storage mechanism are you using? Go to admin -> Plugins -> Caches and run the performance tests. I you have slow disc, then the default caching to disc will be a disaster.

Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hi Tim and Micheal,

Thanks for the responses. I agree that hosting Ubuntu VMs on windows is not a great idea but unfortunately, something beyond my scope.

Also, I tend to agree that disk io is killing. I am planning to use new relic(got green signal for that!) and run tests as suggested by tim tomorrow. I will make sure to post the result.

For moodle data directory, they have a shared drive which I am mounting on my VMs. This is again something the client wants and I am not sure if this is a factor.

Tim, I am using moodle 2.7.



I will post results for performance tests and info tomorrow as well run sysbench for i/o,

Thanks and will keep posted.

Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hi Guys,

I managed to get a screenshot of the performance logs displayed in footer of moodle.

Attaching it. The page is the landing page as seen by admin.

I will post the same for a student ASAP.

I also ran sysbench file io tests and I got transfer rate of about 36MB/s.

Also, using firebug, I can see it waits a lot and apache has a high waiting time for serving the requests.

However, the css and images come very fast. The php files have a high waiting time.

Attaching a screen shot for that too.

Attachment firebug.png
Attachment per-info.png
Average of ratings: -
In reply to rushabh hathi

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Right, so it is not the Database, but it is the PHP code, not all the CSS and JavaScript.

If you want to dig further, this blog post I wrote might help: http://tjhunt.blogspot.co.uk/2013/05/performance-testing-moodle.html

Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

HI Tim,

Sorry for the late response. I saw the blog and would try it.

In the mean time, I wanted to confirm that can this be because of NTLM SSO being ON ? 

I have integrated NTLM SSO using samba winbind service as described here:

http://docs.moodle.org/24/en/NTLM_authentication#Using_the_NTLM_part_of_Samba_for_Apache_on_Linux


Can it be the case that in each page load it is asking for authentication?

I know its a far fetched idea but just wanted to confirm. I am not able to wrap my head around on what is it that apache is waiting for ?


Also you mention the following line in your blog:

Performance advice: if you are running Moodle 2.4 with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcache instead.


Can you elaborate as I am using a load balancer for web sever as well as database.


Average of ratings: -
In reply to Tim Hunt

Re: Troubleshooting performance : what does the following mean : ticks: 69 user: 52 sys: 3 cuser: 0 csys: 0 ?

by rushabh hathi -

Hey Guys,

Just to update everyone, I figured the issue. It was pretty silly of me.

The moodle data directory which I was using was being accessed through LAN and was mounted on my VM.

Apparently that was slowing things down as even cache was maintained in it. I tried using memcache but no great results.

I am now thinking to switching to NFS using an instance of VM.

Will keep you guys posted.


Average of ratings: -