thanks for trying that. The messages that you quoted are actually just "warnings" and they are not causing the final abort.
That fact that this issue only occurs on your largest course is a further indication that the main cause of this problem is either (a) insufficient execution time for the PHP scripts, or (b) there is insufficient memory for zipping process.
You can increase execution time with a setting in the PHP settings file, "php.ini".
- Moodle docs: How to change max execution time
Running the backup from the CLI (command line interface) may get also arround the limitation on execution time.
- Moodle docs: Cron with Unix or Linux
As for increasing the memory limit, you can probably only do that by paying more money to your hosting company