Reduce bandwidth by 80-85%!!!

Reduce bandwidth by 80-85%!!!

by Dakota Duff -
Number of replies: 10

Last night I completed a bandwidth trifecta that reduced the overall bandwidth uses on three different types of pages to an average of 17% of what they'd been!

The first, and most obvious step is GZIP-ing pages, and this is where you'll see your biggest gain.

There are a few ways to do this, but the only one that works for me with my shared host is to enable ZLIB compression in the php.ini file. First, add this line to enable ZLIB compression:

zlib.output_compression = On

Then add this line to determine the amount of compression:

zlib.output_compression_level = 6

The value needs to be between one and ten. I chose six because it seemed to be a popular choice around the web. Choosing one still yields great results and would put less strain on your server.

It's worth noting that in my situation you must have a php.ini file in EACH DIRECTORY you wish to use compression in, so I used another script to create symbolic links back to the php.ini file in my root directory.

Next, you need to tell your server to not only compress PHP output, but to compress CSS and and Javascript, as well. I'm running Apache, so I did this by adding the following line to my .htaccess file:

AddHandler application/x-httpd-php5 .php .css .js

I had done this before and while it resulted in smaller files it also resulted in a longer load time. The culprit was /lib/javascript-mod.php. The problem was that the file tries to specify the content length header manually, so when compression is used the browser doesn't think it's reached the end of the file and it hangs, and hangs, and hangs... until it eventually times out. I fixed this by simply removing the content-length header directive from the file altogether. So you might want to keep your eyes peeled for other scripts that try to force content length.

Anyway, we've now got compression up and running on all PHP, CSS, and Javascript files. But now the CSS and Javascript are now being returned as an incorrect content-type, which may cause some weird behavior. Plus, our CSS and JS files are STILL more bloated than they need to be.

Once again, PHP comes to the rescue. We're going to add a small piece that will run immediately prior to the processing of a PHP page to determine what kind of file we're dealing with and how to handle it. To do this we will add a file I created called prepend-moodle-1.0.php. In my case I put it in an includes folder at my web root (I'm applying these compressions to all pages I server, inside and outside of Moodle). If you only use Moodle, the /local directory would be a good place.

Once you decide on the location, add to your php.ini file:

auto_prepend_file = /full/path/to/prepend-moodle.php

For .js, .css, and styles.php (Moodle's composite stylesheets) files we:

  1. Send the correct Content-type header
  2. Set cache controls and expiration dates and
  3. Minify the output (remove excess whitespace, comments, and other unnecessary characters) *
  4. Rewrite the minified page with new headers

* The methods used for minifying require PHP5

Version 1.0 of my prepend-moodle.php file is attached. Remove the version (-1.0) from the file name before uploading.
jsmin can be obtained at http://code.google.com/p/jsmin-php/
cssmin can be obtained at http://code.google.com/p/cssmin/

Average of ratings: Useful (1)
In reply to Dakota Duff

Re: Reduce bandwidth by 80-85%!!!

by Gary Anderson -
Dakota:

This seems very promising.

One thing to make sure that you do is to test this under the load of many users. It seems like you have considered this, but it is something that many neglect when the do active compression (including attempts at this site, etc.) When things get busy, you don't want the server CPUs to be spending too much time on compression (especially when it is on redundant files) which is why some of these strategies work for single users but don't always work in production.

Another area to optimize is that the default style sheets also expire after 10 minutes, which seems fine for development but means a reload much too frequently in production use. I think setting this for more than 24 hours would be better. There also seems to be some caching issues on browsers when ending style sheets files in .php rather than .css, although there are benefits if switching between style sheets. Mostly this affects whether you need to do a command refresh or a normal refresh. Firebug helps to see this issue on reloads.

In any case, thanks for your contribution on this. I am sure that many of us will want to check out the ideas to see how they work in practice, and your approach seems original.

--Gary
In reply to Gary Anderson

Re: Reduce bandwidth by 80-85%!!!

by Dakota Duff -

Excellent points about server loads. We won't get a heavy user load for another month or so.

I'm not sure what you mean about the default style sheets expiring after 10 minutes. If it's the styles.php style sheets you're referring to, this script changes the expiration date to +3 days.

In reply to Dakota Duff

Re: Reduce bandwidth by 80-85%!!!

by Gary Anderson -
Hi Dakota:

The 10 minutes I was referring to is the existing default cache setting for the standard style sheet. I have posted a bug report to the tracker suggesting changing it to 3+ days as you recommend MDL-15629.

From my observation in firebug, this 125K pseudo-file takes up way too much bandwidth, and if it does not need to be downloaded as often will reduce the compression demands on the server.

--Gary


In reply to Dakota Duff

Odp: Reduce bandwidth by 80-85%!!!

by Adam F -

This is great, thanks!

How to add SVG to it?

In reply to Adam F

Odp: Reduce bandwidth by 80-85%!!!

by Adam F -

I didn't found the way to edit my post so I write new one.


Here is bash script I used to put proper php.ini on every folder in noodle & moodledata trees. (For now I have moodledata inside moodle folder) 


find <path to moodle folder> -type d -exec cp -f  <path to php.ini> '{}' \;


I also attach complete php.ini file I use.

In reply to Adam F

Re: Odp: Reduce bandwidth by 80-85%!!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I just wanted to warn anybody reading this that a very old post had been dredged up and the content probably no longer applies. 

In reply to Dakota Duff

Re: Reduce bandwidth by 80-85%!!!

by Will Manker -

Hi Dakota, 

This was a very interesting read. I didn't know it was this easy to speed up your website this quickly. I have question. I usually use one of the online compressors (I use this one for CSS files and this one for JS files) to pre-compress these type of files. Is it still beneficial to use these compressors or can I stop using them now?

In reply to Dakota Duff

Re: Reduce bandwidth by 80-85%!!!

by James McLean -

How exactly does compressing the PHP pages with PHP and zlib.output_compression differ from compressing them with Apache, using mod_deflate?

Have you done any specific load-testing against these different configurations?

In reply to James McLean

Re: Reduce bandwidth by 80-85%!!!

by Ken Task -
Picture of Particularly helpful Moodlers

@James ... this discusson you've found began in 2008.   No Moodle 2 at that time.

However ... for additional info in your quest to find out:

PHP's zlib output_compression will only work files passed through the PHP handler (i.e. .php files), but Apache's mod_deflate can work on any files (eg static CSS or JS).

Besides that, Moodle is using cache + recommend using Zend OpCache or APC for PHP code.

If Moodle site using a lot of locally served audio/video then think mod_deflate would help more than PHP's zlib.

http://www.whatsmyip.org/http-compression-test/

Run CentOS servers myself and by default, they have deflate turned on.   The PHP required to run 2.7+ of Moodle has to have support for PHP opcache of some sort ... APC or ZendOpcache.

Experimented with an extension for Apache called pagespeed for a while.   Using their test page the site passed with flying colors ... hmmmm, why is that not surprising. ;)

https://developers.google.com/speed/pagespeed/insights/

'spirit of sharing', Ken


In reply to Ken Task

Re: Reduce bandwidth by 80-85%!!!

by James McLean -

So it did, thanks Ken. I only replied because I'd seen others reply recently - I didn't check the date of the original post.

Those were the conclusions we'd come to and that's why we're running mod_deflate (plus APC prior to 2.6, and now OpCache) - I was just wondering if there was some silver bullet with zlib compression in PHP; but that's clearly not the case..