VPS is designed to sustain high load(depends on configuration) and so if your other WordPress isn't using too much of server's resources then "Why not to install another site on the same server?"
You can install moodle and keep both easily.
You can just create a new folder into "var/www/mynewsite" and add your new domain to httpd.conf
now you are free to use "mynewsite" folder with your new domain. now you can install moodle here without any issue. It won't affect anything.
But for safety, as you are going to use same database user(you can also create a new one), better to keep a copy of your Wordpress database, in case of anything wrong occurs(you can take help from this article about How to backup WordPress database)
Thanks