I have the same problem in 1.8 and it has caused a lot of confusion for students and would love to find a solution.
The best I have been able to do is turn on the e-mail based self-registration. This for some reason will direct them back to the course once they pay. It's not a great solution though because I do lose some students when they either do not want to wait for the e-mail to get to them, never get the e-mail because of Spam filters, etc. I would prefer to not use the e-mail self-registration and just have them put in their profile info., pay for the course and have the course open up for them like it used to in 1.6.
Hopefully someone will have a work around for this.
Brett