I had this issue very recently with a new installation - the course was open to all, but the forums requested the user to be enrolled. Only they couldn't enrol because they entered a fault in the space-time continuum and looped round the same screens.
I found the reason to be because the users were defined as having both student AND guest roles. Removing the users from the guest role (thereby leaving them as students only) solved the issue.
Cheers,
David.