I have had problems in the past with an old version (1.5? 1.4) version of moodle and lots of quiz questions and categories.
You may find that upgrading moodle (after taking all precautions) helps. On the other hand others have mentioned that backups started stalling after the upgrade to 1.6.
This will probably depend on the reason for stall. It soulds like the usual lack of memory/php execution time.
As I just advised eslewhere changing to internal zip/unzip routines, if you are not using them already, can change your backup and restore life enourmously for the better. If you have not set the path to the internal zip/unzip routines then that is the first choice.
After you have looked into increasing execution times and memory limits (perhaps these may even be .htaccess settable) you can
backup your course in bits. E.g. backup works without the tests. Then backup half of the tests and then the other half.
By the way, you should be able to upload your course backups to the new server by ftp:
1) upload an empty text file to the site or course files. This will get moodle to create a folder in moodledata called "1", "2", "3" or whatever corresponding to the id of the course - "1" if it is the site files.
2) Upload the backup into the folder ("1","2", or "3" etc) in your moodle data folder using a ftp client in the same way that you uploaded moodle itself.
3) Return to the moodle GUI and restore the uploaded file.
Tim