I'm fairly new to Moodle and have been beating my head against the wall on this one!
We have a Moodle 3.7.1 housed on Microsoft web server 2016 and SQL server 2016 that is having issues connecting to mail server. But here's the rub: that same web server houses another Moodle site, version 3.3.5, that is communicating no problem with the web server (not best practice, I know, but one site is a test and the other is for internal use, if that makes it better).
I've studied and tested the outgoing email configurations countless times and our email server admin has also taken a look and is stumped.
One tip I got was to move over an older email folder to the auth folder but the new site didn't like going backwards.
We'll continue to search in the firewall and email server but could this be a setting buried somewhere on the new site?