Well, first you have to have a limit somewhere. We have courses at the OU with 10000+ students. There is no way that putting that many rows into a single list box would be a good idea. So, there has to be a limit, and that is mostly set to 100.
Oh dear. There is not a single place where you can set this. I was hoping that there would be a single constant in user_selector_base in user/selector/lib.php that would control this, but it seems that each subcass uses its own limit. (Doh! This must be my fault. I implemented this originally. Can't believe I did that. Someone should fix it.)
Anyway, what you should do is expand the 'Search options', and turn on the option "Keep selected users, even if they no longer match the search" then you can search and select lots of users before adding them in batch. You may also find the "If only one user matches the search, select them automatically" option helpful.