Sharing a little more info would help: operating system is? remotely hosted? if so, who is provider? Running apache or other?
Have you checked error logs for apache (assuming you are running apache)?
What does Debugging report when you manually run a backup on the troubled course?
Command line access? Have you tried running the autobackup script from the command line:
cd moodlecode/admin/cli/
php ./automated_backups.php
NOTE: might want to set autobackups in Moodle Admin UI to manual before doing above.
File system of Moodle 2 now interlinked with DB.
What does this query of your DB look like:
select contenthash,filearea,filename,filesize from `mdl_files` where (`filearea` like "backup" and `filename` like "%.mbz%"
Check moodledata/temp/backup/ directory
Are there any .log files larger than 0 bytes? If so, open them with a text editor and inspect. Clues might be provided there.
Are there any 'contenthash' named directories in moodledata/temp/backup/
If so use whatever you can use to browse files and inspect the contents of that directory. View the moodle_backup.xml file with whatever text editor you have online.
In the troubled course, is there any NON-core add-on being used? a mod/block or is the troubled course set to use a customized theme?
'spirit of sharing', Ken