After many hours of frustration, I finally figured out how to set up a Windows 2003 server to allow Moodle to send emails via SMTP. I decided to post some general instructions here for future reference and to save someone the same type of aggravation. I searched everywhere in Google and Moodle.org and noticed many windows users were struggling with this, but most gave up and started using a third party email pluggin as opposed to getting the IIS SMTP to work.
I was trying to get Integrated Windows Authentication through SMTP, because this is how I have been sending emails with other scripts for many years. Apparently, Moodle cannot use encrypted credentials, so the only way to get it working is via No Authentication (which is not an option), or Basic Authentication without encryption (and then create a stripped down windows account just for this).
Here are the steps for the Basic Authentication option:
1-Create a new SMTP virtual server in IIS
2-Under the Access tab in the SMTP properties, click Basic Authentication. Make sure Requires TLS encryption is unchecked.
3-Under Connection Control and Relay Restrictions, add all the local IP addresses (with Only the List below option) on your network that will need access to SMTP
4-Create a new windows user account under computer management-> Local users and groups. Make sure this user belongs only to the Guest group, and remove all rights to remote access, dial-in, and VPN. Strip down the rights as much as possible. Check password never expires.
5-Go back to the SMTP properties, Security tab. Add your windows user account that you just created.
6-Now in the Moodle installation, go to Site Administration, Advanced Features, Plugins, Message Outputs, Email, set the SMTP hosts to localhost (if moodle is on the same server). Set SMTP username and password to the credentials of your new windows guest user.
7-Save Changes and enjoy emails from Moodle.