I know this is an old thread but just had this problem and the solution is within the settings for self-enrollment.
Course Administration --> Users --> Enrollment Methods --> Self Enrollment
Click on the edit icon.
You must ensure that "Use Group Enrollment Keys" is set to YES. By default it is NO.
Hope this is useful.
Dave
