Other courses are smaller then, right?
Check php settings for things like time for a script to run and memory a script can use. Found on the Server -> PHP Info page in Moodle. If time for a script to run is like 30 seconds, increase. Any changes you make to php.ini do require the restart of apache.
you have access to the course still. Go into files, backups to see if there are multiple backups there.
Remove multiple backup files. You are attempting to reduce the size of the backup you are creating.
Check your apache error logs to get a clue.
Turn on debugging in Moodle to see what it might report if anything.
'spirit of sharing', Ken