I did have the same problem.
Sometimes the glossary-entry opens in a java-script-window (yui-window), and sometimes the entry open in a link.
A solutions:
In administration /Plugins /Filters /Common filter settings edit the field “Text cache lifetime“ Choose “No”
Now your Server filter every page and does not use the text-cache. The page will always open in a java-script-window.
The disadvantage: The server needs more time to produce the page.
Background:
Whenever Moodle use the text from the text cache, it does not insert in the header the following line:
Y.use("moodle-mod_glossary-autolinker",function() {M.mod_glossary.init_filter_autolinking({"courseid":"6"});})
Result: the javascript-window does not work.
Without this line the Links do not open in inline-windows.
In my Opinion this is a bug. (but I don't know how to use the Moodle Tracker).
Sorry for the bad english
Greeings from Münster
Thomas