Hi,
It looks like this feature is missing in Moodle 2 even though it's in the Backup and Restore documentation. We just updated from 1.9 a few months ago, so this was an unpleasant surprise for us!
You can vote for it to get fixed (and comment to the developers) here: https://tracker.moodle.org/browse/MDL-30937
A few things to check in the meantime are PHP's memory_limit, max_execution_time and max_input_time in php.ini (there's a little bit about memory_limit here: http://docs.moodle.org/26/en/Performance_recommendations#PHP_performance). Also, if the course is over 2GB, you may run into a limitation in PHP itself (though this may not be happening since you are getting a warning rather than an error). I believe they are addressing the issue of large course backups in Moodle 2.6.
Cheers!
Matt Long