I had this problem, but then found out it was one of the functions returning invalid or not returning.
I changed the return types to defaults, notably i commented out supported_filetypes and supported_returntypes so they used base class values, and it appeared
A lot more to do now though
Btw - in my case i already have a browser interface that is created dynamically via a hotlinked js file, in other applications i just extend tinymce media plugin by overwritting filebrowser option there (loads html page that with <script> and <css> tags pointing to right location, passes back url on selection using tinymce_popup code).
Now i gota find out how to make this work here