Nginx

Nginx

by Frankie Kam -
Number of replies: 12
Picture of Plugin developers

Hi.

I'm currently running Moodle under a Centos VPS, with Apache 2 web server.

I've read about Nginx (http://wiki.nginx.org/Main) and am considering about changing from Apache to Nginx.

Has anyone done a changeover from Apache to Nginx and managed to get his or her Moodle site unscathed from any bugs/meltdowns of the transfer? If so, how can I do this BY MYSELF with the least amount of stress or white-hairs?

Any wikis / step-by-step tutorials that you can recommend for me to do the full setup of Moodle on Nginx on a VPS that currently runs on Apache and Moodle 1.9.7?

Yup definitely asking for the moon here.

Hope this helps (me).  big grin
Frankie Kam

Average of ratings: -
In reply to Frankie Kam

Re: Nginx

by Ruslan Kabalin -
Picture of Core developers Picture of Moodle HQ Picture of Moodle Workplace team Picture of Peer reviewers Picture of Plugin developers

Hi Frankie, Apache and Nginx are not comparable that way. It is important to understand that, though Nginx is known for better perfomance than Apache on static content (e.g. images, css), it is not capable to serve dynamic content directly (e.g. PHP). To server dynamic stuff, it may use FastCGI, or act as a frontend for apache, so that static stuff is served by Nginx, but all dynamic request are handed over to Apache (IMHO this is the better way of using Nginx for PHP). You may see this benchmarks or this to get an impression that Ngnix is not that fast when fastcgi is used for php.

Applying this to Moodle, if you have lots of static stuff (images) on your pages, you will gain some extra performance by setting up Nginx as frontend for static content (and apache as backend for dynamic stuff) and tuning everything up properly, but do not expect any magic ;)

Average of ratings: Useful (1)
In reply to Ruslan Kabalin

Re: Nginx

by Frankie Kam -
Picture of Plugin developers

Hi Ruslan

Thanks for the quick reply.

So, it's Nginx for front end static content (images, css)
and
Apache for backend dynamic pages (PHP).

Got it.

Well, I just only (yesterday) signed up for another VPS with my webhost
so that I can experiment with Nginx. I hoped to setup Nginx, eaccelerator and Varnish, and then Moodle on my 2nd VPS by referring to this link:

http://www.travisberry.com/2010/10/lightning-fast-php-server-with-nginx-eaccelerator-and-varnish/

Didn't want to break my Moodle site which is found in my first VPS at http://scm.moodleace.com.

Now to find some stuff on how to setup Nginx to serve front end
static content...

Frankie

Average of ratings: -
In reply to Frankie Kam

Re: Nginx

by Greg Lund-Chaix -

If you have Varnish in the mix, nginx isn't going to make much of a difference (if any).  The benefit of nginx (or Varnish, for that matter) is on static content - of which Moodle produces *very* little.  You'll be optimizing the server for the stuff it's already not working hard on (static content) without helping it with the stuff that *is* resource-intensive (PHP).

I use nginx for routing SSL traffic from my front-end cache/load balancer (running Varnish for non-SSL traffic) to the backend Apache nodes. It does a fine job at that, but I've yet to see a compelling argument for replacing Apache with nginx for a Moodle server.

Average of ratings: -
In reply to Greg Lund-Chaix

Re: Nginx

by Frankie Kam -
Picture of Plugin developers

Ok Greg, thanks for the headsup.

Erm what about this (Nginx + PHP5 + FastCGI + xCache on Ubuntu?): http://www.howtoforge.com/nginx_php5_fast_cgi_xcache_ubuntu7.04

and this?

http://lucasforge.2bopen.org/2009/09/caching-dynamic-content-using-nginx/

and this

It's possible to cache the dynamic content with nginx, by using proxy_cache_path, proxy_cache, proxy_cache_valid etc.
Supports file based and memcached based caching.

file based example:
proxy_cache_path
/some/directory levels=1:2
keys_zone=one:10m;

which is taken from: http://www.debian-administration.org/article/649/Speeding_up_dynamic_websites_via_an_nginx_proxy

?

Frankie Kam

Average of ratings: -
In reply to Frankie Kam

Re: Nginx

by Greg Lund-Chaix -

I guess I'm a little confused at what you're trying to accomplish.  Is your current Apache + eAccelerator config not able to handle the traffic on your site?  If it's providing acceptable response times, what is the purpose for doing all the extra work adding nginx into the mix?

Don't get me wrong, I like nginx very much and use it in situations that require a high volume (usually of mostly static content).  But nginx is not necessarily a good fit in cases like Moodle where the vast majority is dynamically generated content coming from PHP.  Trying to graft nginx into a Moodle system seems to me to be a (possibly ineffective) solution in search of a problem.

"It's possible to cache the dynamic content with nginx, by using proxy_cache_path, proxy_cache, proxy_cache_valid etc."

This sounds like a very bad idea to me.  Dynamic content is just that - dynamic.  You don't want to cache it.  If you cache it, you risk the user getting a stale version of the page.  As an example: a teacher modifies a Moodle course, saves the changes and reloads the page.  If you've set the server up to cache dynamic content, they're likely to receive the cached version of the page without their changes.  Teacher gets mad that their work was "lost", re-does them, and re-submits the changes.  Repeat until the cached version of the page expires or is purged.  Suddenly all the changes appear, possibly miltiple times over.  You now have a very confused and frustraded teacher who just wasted a lot of time.  Caching dynamic content is a very dangerous place to go unless the backend application (Moodle in this case) is aware of and can control the cache, flushing outdated content from the cache when content on the backend is modified.  Moodle doesn't do any sort of proactive cache flush on update activity (nor is that really practical, since the methods vary widely depending on what sort of cache is used).

I use Varnish in front of my Moodle servers to cache as much of the static content as I can (providing the benefits I think you're trying to accomplish with nginx).  However, I have it set to bypass the cache for the vast majority of the Moodle traffic because Moodle dynamically generates nearly all the content delivered to the browser.  A very small percentage of the content delivered on a pageload (primarily JS and theme images) is served from cache, but the rest has to be delivered from PHP.  Since PHP with nginx doesn't really allow for a shared opcode cache, it's likely it'll actually be slower than a comparable system with Apache and APC or eAccelerator.

My suggestions:

  1. Find where your bottlenecks are before you start working on optimization.  You can't hit a target without knowing what you're aiming at.  You mentioned elsewhere that your server is on somewhat limited bandwidth (512Kbs?).  I expect your VPS can probably saturate that pipe easily even with a non-optimal Apache config.  Putting a bunch of work into tweaking your HTTP daemon to squeeze a little more performance out of it is going to be meaningless if the server doesn't have enough bandwidth to support the additional traffic.  When you're doing load testing, watch the disk I/O, RAM and CPU utilization on the server.  That should help you figure out whether you're processor-bound, short on RAM, or I/O-bound.  If none of those three seem to be the bottleneck, it's quite possible you're bandwidth-restricted and no amount of server-side tweaking will help.
  2. If you decide you need to add caching to help reduce the concurrent number of httpd threads (and therefore the amount of CPU time and RAM needed) or disk I/O, choose one tool for your cache - Varnish, Squid, or nginx (configured as a reverse proxy cache).  Using more than one just adds complexity without additional benefits.  I recommend Varnish personally, but nginx is quite capable of doing an excellent job in this role as well.
  3. Keep Apache as your HTTP daemon.  Moodle is built with the expectation that it's running on Apache and the changes you'd have to make to run Moodle with an alternative HTTP daemon are probably more trouble than they're worth unless you're very short on server RAM (which isn't as likely given the 512MB number you mentioned).
Average of ratings: Useful (2)
In reply to Greg Lund-Chaix

Re: Nginx

by Frankie Kam -
Picture of Plugin developers

Hi Greg.

Guess I got carried away by the rush I felt when I eAccelerated my Moodle site, plus mod_deflate on-the-fly compression and even that Apache header expires thingy. And YSlow Yahoo optimsation best practices. I felt the need..the need for speed!!! And more speed please.  Like the Titanic, I wanted to go faster. Why because I want to see a day when 25 students logon to http://www.moodleace.com, take a 60 question-quiz and not see a single lagged white screen showup. Heh.

So when I read about Nginx, I thought wow, goshdarn...those Russians, first the T34, then AK47, then the Hind, SU-27 1.5 Flanker flight sim, then IL2 Flight Sim and then now Nginx....minimal specs, crude but effective and what awesome performance!

I read about Nginx being anti-bloatware, running on some Megabytes against Apache's using 300Mb RAM for multiple threads, etc.. the whole hype. Was it really all just hype?

Well I wanted to try it out for myself. You know, set up a lean-mean-fast-nginx machine with Moodle running on it. Pit http://www.moodlefreak.com against http://www.moodleace.com in a showdown of optimisation server benchmark data. And then finally, announced to the civilised world Moodle.org, that, hey it's not hype after all. This thing really rocks Apache's world.

Anyway, your explanation is very informative, logical and makes lots of sense really. Why would anyone in their good mind want to cache dynamic data and mess up the teacher's mind and sanity? Am I convinced of your learned arguments? Impressed yes.

Convinced? Well I should be by now..but.., I've already rented one month's worth of a new VPS and I'm willing to go the full mont...erm.. full hog to install an alernative Moodle setup - against the better advice of the more experienced ones.

If I succeed in getting "nginxed" http://www.moodlefreak.com to outperform "LAMP" http://www.moodleace.com, then GREAT!! Your reading all this, and you're probably saying "Well, good luck with that!".

However, If I fail big time, then in the spirit of Thomas Edison, I didn't fail, in fact I learnt so much more about servers, VPS, what makes Moodle tick and how to make it go faster or in the end, what doesn't make it go any faster. And my 2nd VPS becomes the proverbial sacrificial goat.

I also have this very useful VPS tool: backup and restore of VPS container. So if I make a mess out of the server and corrupt it outright, to the core...I can then restore the container to its previous working state. Lastly, I can cancell that VPS subscription within the first 30 days right? At least that's what the legal stuff says on my Web Host's website. Now you know where I am coming from. So is there "method to my madness" then?

Anyway again, here's my Munin images of my first VSP http://scm.moodleace.com, the one I am trying to tweak so that it can outrun the Titanic - in all its glory. I've LightGalleried them here. Guest access allowed. But not for long...maybe just for 1 week. What do you make out of them images, Greg?
http://scm.moodleace.com/mod/lightboxgallery/view.php?id=2017

You're looking into the very soul of my Moodle VPS. All its glory and warts, but for the love of me, they're all colourful pictures to me - it's all Greek to me. Greg, can you tell me the story that those graphs, charts and figures are telling you?

Frankie Kamsky

Average of ratings: -
In reply to Frankie Kam

Re: Nginx

by Greg Lund-Chaix -

"Was it really all just hype?"

No, nginx really is blindingly-fast when used appropriately.  It's a lean, mean, bit-serving machine that can crank out an astounding amount of content with a very small memory/CPU footprint.  I heartily recommend using it where it makes sense.  But, to use your Titanic example, you need to know if tweaking your HTTP daemon is actually turning the ship away from the iceberg, or simply re-arranging the deck chairs or telling the band to play a different tune.  Until you identify where your bottlenecks are, you're going to be flailing around ineffectively in the dark.  It's great that you're not afraid to fail, but you need to stop twisting knobs randomly on the machine and figure out which knobs need to be turned to improve your system's performance.

Looking at the Munin images you posted, I have a question and few observations:

  1. The question: are these images from when the system is under load?  I.E., is it showing a time when you have those 25 students hitting the quiz all at once?  If not, they're meaningless.  You need to see what's happeniing when the system is working hard, not when it's sitting idle. My ovservations below are predicated on the assumption that the system is actually under load.  If it was idle, stop reading here and start gathering data when the system is actually struggling to serve user requests.
  2. Four.jpg: you're not CPU-bound.  At least not when these images were taken.  All that yellow is idle CPU.  On the week view I see a couple of blips where the equivalent of one of the (apparently four) available CPUs is fully utilized, but the other three are snoozing away in idle dreamland.
  3. Seven.jpg: the resolution makes it a little hard to read the memory graphs, but it looks like you're not swapping at all.  That's good - it means you're probably not running short on RAM.
  4. Six.jpg: load averages look good.  If the load average starts showing significant amounts of time above the number of CPUs (it looks like four in your case), you should worry.  It's not.  It spiked to 3 once - no reason to be concerned.  I've had sites still functioning reasonably well with a load average at twice or three times the number of CPUs.  So on a 4 CPU box, any load average of less than 4 is perfectly fine.
  5. Given the above, I don't see much of anything on the graphs that would indicate a RAM or CPU bottleneck.  Disk I/O issues are unlikely - we would see a spike in the load averages if disk I/O was the limitation.  If the system is really under load when these graphs were taken, my initial impression is that you're looking at bandwidth being the limiting factor, not anything server-side.

Munin is good to look back at what the system has been doing in the past, but for troubleshooting performance issues you really need to be using tools that report the system's status in realtime while the system is under load.  Here are some command line tools that you can on the server that might help figure out where your bottlenecks are.  Run these from the shell while the server is under heavy load and you should see some indications of where your bottlenecks are:

  1. top - top should already be installed on your system.  Run it from the shell and it'll give you a good idea what your system is doing.  Pay attention to the  "Cpu(s)" line in the header.  It'll look something like this: "Cpu(s): 47.7%us,  2.5%sy,  0.0%ni, 48.8%id,  0.2%wa,  0.2%hi,  0.6%si,  0.0%st".  The two most useful columns are the "us" (the actual CPU used by processes like httpd and mysqld) and "wa" (processes waiting on disk I/O).  The list of processes below will tell you which processes are using the most CPU, RAM, etc.
  2. free - like top, it should be already installed.  It gives you the total and currently free RAM on the system and whether or not you're using swap.  If you're using swap, you're short on RAM and you should look at either adding RAM or reducing the RAM needed (either through caching or tuning your httpd).
  3. mtop - http://sourceforge.net/projects/mtop/ - basically "top" for MySQL.  It'll help tell you if your MySQL daemon is struggling.   Look for a high cache hit ratio and a low number of threads running.  If a large number of threads is building up, your database server is likely a bottleneck.
Average of ratings: Useful (1)
In reply to Greg Lund-Chaix

Re: Nginx

by Frankie Kam -
Picture of Plugin developers

Greg

Many thanks for your comprehensive replies!!!

I've used top, free and mtop. Although I don't know how to use mtop to flag tables where bottlenecks  could be solved by creating a MySQL index.

I've got a Question.

From http://royal.pingdom.com/2010/02/23/nginx-the-little-russian-web-server-taking-on-the-giants/

>I was reaching capacity of my 1gb VPS RAM quota running Apache2, PHP,
>PostgreSQL. There were 40 apache processes each weighing 15-20mb
>averaging 600-700mb RAM.
>
>I just installed NGINX in front of Apache in a reverse proxy config. NGINX >serves all static content (JS, CSS, Images) and proxies PHP requests to >Apache/mod_php.
>

How do I do this?


>
>Its amazing how efficient this setup is!
>
>Server now has just 15 * 15mb Apache processes (average 225mb RAM) and >4 * NGINX processes (3mb RAM).
>
>Net result: capacity for 3x as many concurrent requests / vistors!
>

Now that looks like what I could want. Of course like you said, my measly 512Kbs Internet link could be the bottleneck as it can be saturated with 10 or so users taking quizzes, etc.

One of the things that set me off on my Nginx "Quest" was my encounter with http://loadimpact.com. And in fact a graph that the test loading site produced:

I believe my VPS "hung" after 20+ load-stressing clients. I'm giving http://loadimpact.com
another go at my http://scm.moodleace.com. That way when I compile the Munin graphs, they
would have produced under stressful conditions (for the VPS, not me!).

Frankie Kam

Average of ratings: -
In reply to Frankie Kam

Re: Nginx

by Greg Lund-Chaix -

Your enthusiasm is infectious, Frankie, but you really do need to step back and look what actually needs to be done before charging in and making changes.

For example: you said you want to improve the efficiency of your system by reducing the number of Apache threads needed by putting nginx in front of Apache as a reverse proxy cache.  A good idea in the abstract, as it offloads the static content onto the lightweight nginx processes and frees up the big heavy Apache processes for serving up the PHP-generated content.  Unfortunately, this sort of configuration is of limited benefit in a Moodle environment where almost none of the content is static and has to be processed through PHP.  Even some of the CSS in Moodle is non-cachable.  The reverse proxy cache ends up just passing everything on to the Apache backend.  What does this mean in your case?  You're adding another layer of daemons using your scarce server RAM that do nothing but accept the connections and pass them back to your existing Apache daemons - very likely a net loss on a VPS that's short on memory.  It's not the fault of nginx, it's that you're talking about using it in a situation where it's not necessarily appropriate.

> I've used top, free and mtop.

OK.  And what did top tell you when you were hammering away at the server with a load testing tool?  Was the load average high?  Was the processor utilization high?  What about the I/O wait?  Was the system using swap?  How much free memory does it show?  Did mtop show any threads building up?  How many?  What were the queries the threads were executing?

Until you can answer these questions, making arbitrary changes to the server config could easily make things worse, not better.

>Although I don't know how to use mtop to flag tables where bottlenecks  could be solved by creating a MySQL index.

You're better off looking at the MySQL slow queries log and learning how to use the 'EXPLAIN" command.

>>I just installed NGINX in front of Apache in a reverse proxy config. NGINX
>>serves all static content (JS, CSS, Images) and proxies PHP requests to
>>Apache/mod_php.

> How do I do this?

There are a lot of howtos on using nginx as a reverse proxy.  A Google search on "nginx reverse proxy" should give you more info than you need.  But as I said above, I don't know if this will actually get you much, if any, improvement on your Moodle system.

> Net result: capacity for 3x as many concurrent requests / vistors!

For sites with quite a bit of static content, this is very likely true.  A Drupal site with a lot of anonymous users can easily see a tenfold or hundredfold increase in the number of concurrent connections when using a reverse proxy cache - I've done benchmarks that show exactly that sort of increase.  For Moodle sites where all the traffic is authenticated users and even the CSS is not static, not so much.

-Greg

Average of ratings: -
In reply to Greg Lund-Chaix

Re: Nginx

by Frankie Kam -
Picture of Plugin developers

Greg

I'm actually quite pleased with my site's current performance (with eaccelerator and mod_deflate helming the speed issues) with a small class of 10 to 15 students. Next semester I have 25 students per class, and so I am keen to get the site optimised in terms of speed and performance to handle that number of concurrent students without too many complaints of lag and timeouts, etc.

Okay, so for TODAY, I've decided to step back and get back to the basics. Optimising my Apache, PHP.ini and MySQL settings. One step backwards today...perhaps tomorrow, TWO steps forward.

If you don't mind, perhaps I could invite you to take a look at my Apace, PHP and MySQL settings in the 3 Wikis and I webpage of my PHP Info settings that I have set up at:
http://scm.moodleace.com/course/view.php?id=52

If i could beseech you to spend a couple of tens of minutes of your precious time to check my settings for:
1. Glaring errors
2. Redundancy
3. Overstated or understated values
or anything that could be changed in order to improve the performance of my Tamagochi (I mean my virtual pet toy - that is my Moodle site!), I would be every so thankful.

I have spent countless hours surfing around to read webpages that if I follow correctly, will allow me to optimise my site. The problem is...the MORE I READ the LESS I KNOW if I am doing the right thing.

The course allows for Guest access. My system specs: VPS 512Mb RAM, CentOS release 5.5 (Final), Apache 2.2.3,PHP Version 5.2.17 and MySQL.

Frankie Kam

Average of ratings: -
In reply to Greg Lund-Chaix

Re: Nginx

by Jürgen Strobel -

Greg: This proxy setup is effective, but not primarily because nginx can serve static files directly. As you have noticed this does little for highly dynamic apps.

The reason is, in this setup apache/modphp can deliver its result almost immediately to nginx, which will take the job of spooling it slowly to a remote client. While nginx is delivering, the apache+php interpreter process can be freed up and made to work on the next request again, thus not holding all its resources during network delivery. This is worth the extra layer for all but the smallest sites, and scales far better than a setup where content creation and delivery is bound to the same process and the same tuning parameters.

Frankie: I suspect you can improve your setup by changing your process limits with these insights.

1. You want only a few apache/php processes, like 2-4, because now these are essentially CPU-bound (or DB-bound which is the same for a sufficiently small DB and typical queries). This will prevent cache trashing, and the front end nginx will serialize access to these backends. Php processes don't have to stay around during nginx's delivery phase, and you don't want more than neccessary. On a multi-core machine around your number of cores, but minimally 2-4.

2. You want as many nginx processes as you can get to saturate bandwith and/or RAM. For sure more than 4, and much more than your number of apache processes. You have to experiment here. With only 4 I suspect that with big traffic spikes some of your users could get server not available errors.

You can achieve the same effect by using any other 2-tier setup, like apache (without modphp) + fast-cgi, or even 2 apache instances one of them acting as light reverse proxy the other as php engine. I reccomend fast-cgi because it is quite simply, highly scaleable, and best of all, provides extra security by allowing to run fast-cgi processes as a user different from apache or www-data easily, all without having to learn and manage another web server.

-Jürgen

Average of ratings: -