it can happen when the course is too big. You can try to make the backup of this course in command line. The script is in the admin/cli directory.
If it's also possible activate the debug to see the exact error.
And somtime even if the UI display a timeout the backup is restored.
You can also activate the backup/restore asynchronus it maybe also avoid the error.