I think it has more to do with the particular page than with server performance. Some pages in moodle load more than others even with the extensive caching. So it may take the browser some time to process everything.
I too consider that an issue. Not everyone agrees though.
Usability-wise I think that editors display mode should be made more flexible. Many cases of text input do not require the full blown editor. I would prefer to be able to set the default to open a textarea and where needed click to open an editor for the textarea I'm currently on (the format value which accompanies every editor may be used to make the preference sticky).