Consider this response as 'insight' (kinda) as a former mail server(s) [note that's plural] and mail gateway(s) admin ... which in itself was a full time job which my employer [at the time] did not consider to be 'full time' - just 'another hat' under 'internet'.
Using SMTP relay for anything, let alone moodle, is more complicated than moodle docs/config and the other end (the smtp server doing the relay) leads you to believe.
The following links might help ... and they are technical ... can't avoid that. So with that in mind:
Please see:
https://support.google.com/a/answer/2618874?hl=en
https://support.google.com/a/answer/6140680?hl=en#zippy=
also
https://support.google.com/a/answer/2956491?hl=en&ref_topic=2921034
also
https://support.google.com/a/answer/178333?hl=en&ref_topic=2921034
Personally, have found using Sendgrid's API smtp relay to be the 'easiest' and most reliable with reservation as to volume of forum mail. Note: there will be differences of opinion using Postfix below.
smarthost for Postfix installed on same server as the Moodle as local only (sending only)
https://support.google.com/a/answer/2956491?hl=en&ref_topic=2921034#zippy=%2Cpostfix
https://sendgrid.com/docs/for-developers/sending-email/api-getting-started/
With google, do you have Google Oauth2 config'd in your Moodle? How about Google Drive/Docs?
The requirements to get those to work carry over to Google as SMTP relay.
Now if you host on shared hosting ... you can forget all above. If you are on a self-managed VPS and have ability to sudo to root + are not adverse to command line, then there's a greater chance you will succeed in getting this fixed and working.
One thing you should probably do is to attempt to find out just how much forums and anything in your Moodle would use email notifications. There is no admin report for that.
Why? Volume of messages = cost. Many (if not all) SMTP relay offered as a service does have a cap at the free level and going over that cap will involve a fee.
'Spirit of Sharing' (SoS), Ken