Trying to run redhat 5.2 would surely reduce the requirements of CPU, hard disk and memory (for example, Red Hat 6.2 only needed 16MB of RAM to run). But there are big problems involved in doing that
1) Red Hat does not support any system before 7.1, and the normal way to keep systems upgraded is applying patches until a new version is released and then upgrading to that version. This is very important, since it is critical for a server to have its operating system up to date. If not, you will find it quickly cracked, and it is likely that you loose all your data.
2) You must make sure that you meet the software requirements, specially having PHP 4.1.0 or upper version, Apache (still better 1.3.x, since 4.0 has still some problems with PHP) and MySQL. All these applications have to be updated also, for the same reason, and they do get resources.
So, I would better try to reduce the Red Hat services to the most, deactivating everything that is not really necesary and unistalling X Window, KDE and Gnome (they take most of the resources).
You can also try to find another Linux distribution
, there are some that fit in even in a single floppy! The critical point there is to assure that they are mantained and you can keep them up to date, and also run latest versions of Apache, MySQL and PHP.
You can also try FreeBSD
, as John is doing. It is a very reliable operating system, similar to Linux (I think that Yahoo uses it for their servers). I think it also takes less resources and there are also mini versions (perhaps picoBSD, miniBSD,...?).
You have plenty of options to choose from: buying a new computer, hosting, optimizing your server, finding a small Linux, trying FreeBSD, keeping slowlyness,... It is up to you. I hope you improve your Moodle service. If so, please let us now, to learn from your experience.