Hi John,
IMO installing postfix will do nothing for you other than add unnecessary complexity to your setup. You do not need an SMTP/MTA server running locally if your Moodle is only going to be sending email (lets ignore for now that moodle can receive emails too...). PHP can send emails just fine without it.
You're problem, as Ken discussed, is probably that that your home webserver looks really spammy to other receiving MTA. This is what I believe are you best options to explore...
First make sure your moodle is sending emails as a real rotatable domain, nothing says spam like a fake sender address., see,
Site administration > Server > Support contact > Support email
and
Site administration > Plugins > Message outputs > Email > No-reply address
Then either....
1) Use an upstream SMTP server to relay your mail. A lot of ISP's will provide this functionality for you and you just need to add the appropriate details into moodle (Site administration > Plugins > Message outputs > Email). Depending on the service your ISP provides you may need to ensure moodle is sending as a valid email address (see above).
If you have set this up and it is still not working, enable SMTP debugging,
Site administration > Development > Debugging > Debug email sending
And then try a 'forgotten password' process from your login pages, you should see the full output on the SMTP conversation with with the upstream SMTP relay server which will usually tell you exactly what happened. Also check the moodle logs in,
Site administration > Reports > Logs
And set the activities column to 'site errors'
2) Make your home server look like a real SMTP server. Ken mentioned SPF, DKIM etc... That is great, but advanced, forget it for now. The most important thing to do is ensure your global IPs forward and reverse DNS are the same. Now this might not be possible if you are on a 'home' DSL type connection and are NAT-ING your servers IP address... If this is the case go back to option (1) as this is never going to work well for you with either PHP Mailer or Postfix.