151 is default for MySQL. Despite other advice, think I'd increase it anyway. And, yes, just like php changes requiring a restart of Apache, any change to my.cnf would require a restart of the mysqld service.
Noticed by the top output that mysql-workbench is eating up the memory. mysqld didn't even show in the list?!!! Using workbench you could be fooling yourself. Ok, fine, use workbench when you are working with MySQL, but shut it down when not. It shows workbench has been running for days!
And another thing with an under-powerd (memory wise) ... the GUI desktop just gets in the way ... and it does eat up memory as well. Notice XOrg is second in the list of top.
One can have the server boot at command line and then, if one ever needs it, one could launch the GUI desktop IF logged on in a terminal session ON the server. BTW, I normally use ssh shells remotely from a workstation. That uses little resources so when I look at top, etc. am seeing things as 'clients' might see them via browser.
On a web server using a resource hungry app like Moodle 2, one needs to shutdown/not use anything except what is necessary ... ie, anything that takes away from PHP, MySQL, or Apache ... the purpose of the server. Of course one leaves alone those things related to the functioning of file system, network, etc..
Yes, increase the memory via VM. Restart the guest OS. Ubuntu should pick up on the new memory and begin to manage memory differently and all by itself.
However, the size of the swap will not increase automatically - but, if you give it more memory and configure apache/mysqld to use it appropriately, one should NOT be needing SWAP space.
Good advice has already been given you concerning apache.
When it comes to a server that is to be the filestorage, the DB, and the web server for Moodle it's a balancing act between the config of the DB and Apache ... both need more than defaults ... even with small Moodle sites due to some activities (like test/quiz) being resource hungry.
'spirit of sharing', Ken