I did have the same problem.
Sometimes the glossary-entry opens in a java-script-window (yui-window), and sometimes the entry open in a link.
In administration /Plugins /Filters /Common filter settings edit the field “Text cache lifetime“ Choose “No”
Now your Server filter every page and does not use the text-cache. The page will always open in a java-script-window.
The disadvantage: The server needs more time to produce the page.
Whenever Moodle use the text from the text cache, it does not insert in the header the following line:
Without this line the Links do not open in inline-windows.
In my Opinion this is a bug. (but I don't know how to use the Moodle Tracker).
Sorry for the bad english
Greeings from Münster