The fact you are saying 'zip' means it's a very old Moodle ... 1.9 vintage? Better backup/compression to .mbz files can in around 2.3 or so + there were CLI only scripts that could do course backups ... taking apache out of the loop.
Confusing ... 'while executing the same script in the command mode' ... hmmmm, customized whatever?
Which then relates to the version of PHP (guessing 5.3) thus also leads to platform ... linux or Windows?
Linux 5.3 PHP zip had a limit even with 86_64 of 4 gig and it was the processes of 'copying' the zip. It was an operating system limitation which could be overcome, but one would have to re-compile the kernel to get the support for sizes over 4 gig.
So if I'm guessing correctly, think the only approach one can take (if stay with what you have) is to increase PHP limits to very high settings .... time for a script to run and memory used + backup courses in pieces ... like you've semi-discovered ... then, if one wants to restore that course to same system, take the two part route ... install one backup and then the other in the same course.
Time to upgrade? All the way round?
So I could be in-correct .... your Moodle version? And OS? and php version?
'spirit of sharing', Ken