Hardware and performance

IMHO your priorities should first be CPU cores and disk performance. More cores and some SSD will return the most gains. I'd hazard a guess that 32GB of RAM would be MORE than enough... just make sure you configure you database process to actually use it.

I am not saying that getting a second server for a database is a bad idea... it's usually a good idea.   But... how did you establish that the database is the "pinch point" on your system? If it isn't then you'll be very disappointed.

If it's a MySQL database, the main benefit of lots of RAM is the ability to configure the database to use it to buffer your data (innodb_buffer_pool_size). Ideally, you want enough RAM to be able to hold your entire database in memory, but once you achieve that, the benefits of adding more RAM will tail off, so that's kind of the sweet spot. The problem with that is that it's not fixed - you need to either build in some spare for the future, or make sure you can add more when your database gets bigger.

Others have given good advice.   Suggest running tuner on your server to see what it might say in a couple of areas:

What does it say for:

[OK] Maximum reached memory usage:
[OK] Maximum possible memory usage: 

and the item the display of the above two ....

[OK] Overall possible memory usage with other process is compatible with memory available

There is also a section in InnoDB Metrics.

Number of users that have accounts doesn't necessarily equate to the number of users actually logged on *and* doing something.  So what other data does one have that indicates splitting out the DB onto another server will increase performance/solve issues?

While there are things one can choose/do to make sure the speed of a dedicated DB server is optimal, that setup still involves networking ... before the application.  That becomes a piece of the puzzle when things go south (if they do go south) and could involve other things not considered.   In the past, as an example,  it was thought by some that a fiber channel connection was needed (web -> DB server) to get max speed ... but Cat 6 wire now-a-days thought good enough to handle it ... IF not 20 hops away (as an example).

'spirit of sharing', Ken

Thanks for the replies. I had requested some server optimizations with our dedicated server extended support plan and in the request was to get OPcache enabled (already installed, just needed the lines it added to php.ini to be uncommented and switch to fcgi so it would have a handler it could work with). I got the reply all was done and when the site was still so slow at peak, I felt there was no more that could be done. We did figure out that the OPcache was not yet on. We got it enabled properly last Friday. I had heard that it would speed things up Moodle by 2x to 3x times, but at peak it really feels more like it was a 10x gain. That is one useful tool to get on a busy site!

So, we are keeping up with demand now and will not need to resort to splitting the database out to get a performance boost just yet.

Now to figure out why we are starting to get Internal server error pages here and there for the last few days - One course will not load and a few less important pages are turning up with it. I am beginning to wonder if the server side troubleshooting will ever end. It is a bit like digging in sand. You shovel out one thing and it is replaced by something else. I have really learned a lot since moving to Moodle 2.9 and it has actually generated an interest in learning more about the server side of running a Moodle site - IF I can eek out more time than I have had so that I can learn it right and not just piecemeal here and there at least to get solid foundations under me.   :0)