General help

Well, after a couple of days investigation and testing, I've finally gotten it working. Quite a few variables at play that really confused things!

I imported the cert into the Windows certificate store of the Moodle server as a trusted CA , but this didn't work. I had previously had similar problems with LDAPS, and found documentation about saving the cert of the LDAPS server and adding a bit of extra config, so  assumed I'd need to do something similar with SMTP TLS.

I quickly found reference to the cafile option in php.ini and placed a pem of the self-signed cert in a readable area and pointed php.ini to it, but - fail.

That's when I thought to try connecting to Google's SMTP server to confirm things were otherwise working and was surprised to get the failed STARTTLS, which was unexpected.

After much head scratching, I eventually realised that enabling the cafile option stops openssl/php from using the OS's built-in certificate store. So with cafile enabled, openssl/php stopped trusting Google's CA. I added this cert into the same pem referenced by cafile and could then connect to Google's SMTP server.

But still the self signed cert wouldn't work, despite being in the same pem.

I found reference to openssl's SMTPOptions for verify_peer, verify_peer_name, and allow_self_signed. I added these in to PHPMailer.php and found I could connect to our SMTP server only if I set verify_peer to false. So basically ignoring validation altogether, and certainly not something I wanted to leave.

In one of my later tests, I disabled the cafile (so allowing PHP/openssl to again use the Windows certificate store) and found I could connect with verify_peer true, verify_peer_name false, and allow_self_signed false. So it was doing a bit of cert validation, but not requiring that the cert was issued to the server using it.

As a self-signed windows cert, the 'Subject' of the cert was set to the servername, not it's FQDN. So I changed the connection in Moodle to use just to the servername and could connect with verify_peer true, verify_peer_name true, and allow_self_signed false! These are the default settings, so I could then completely remove the extra SMTPOptions I had added to PHPMailer.php.

This leaves me thinking a couple of things aren't quite working right, so I wanted to summarise here in case anyone else comes across this problem themselves in the future:

1) cafiles doesn't appear to work with self-signed certifcates (or at least not the self-signed cert our Windows Exchange server issued itself). I added this same cert to both the Windows certificate store and to the pem that I had added Google's CA to. The self-signed cert is only accepted when php/openssl is using the Windows certificate store, not when its using cafile; while it accepts the Google cert works in both cases.

2) The 'SMTP Hosts' setting in Moodle must use the name that appears in the 'Subject' of the certificate, even if the name you'd like to use is also in the 'alternative subject names' section of the certificate.