What are the particulars to PHP? 32 bit version? That could be the reason why.
Continuing to attempt might fill up space in /moodledata/temp/backup/ thus compounding the problem. It is safe to manually remove the contents of /moodledata/temp/backup/ as long as automated backups isn't running.
There is in the admin/cli/ directory, a backup.php script that backs up a single course.
Perform backup of the given course.
Options:
--courseid=INTEGER Course ID for backup.
--courseshortname=STRING Course shortname for backup.
--destination=STRING Path where to store backup file. If not set the backup
will be stored within the course backup file area.
-h, --help Print out this help.
Example:
\$sudo -u www-data /usr/bin/php admin/cli/backup.php --courseid=2 --destination=/moodle/backup/
If that also doesn't work, only alternative (with existing setup) is to break apart the backup ... ie do it in as many parts as will work. Can't tell you how many parts or what to include as some activities are 'heavy processing' ... like quizzes. So below, one might have to treat quizzes like files.
backup1.mbz might be nothing but quizzes
backup2.mbz might be half of the files linked.
backup3.mbz might be the other half of the files.
OR ... if your Moodle version (now quite old) will work with it, find a 64 bit version of PHP but ... not sure your older version of Moodle supported.
'spirit of sharing', Ken