I know I'm a bit late to the party, but I had a site with 50k users and upped this limit, no problem...
... until I ran a large report - it wasn't the server that died, it was the browser. Running a full user report ran ok (~2.5 seconds) and it then proceeded to display the whole thing in an html table. Neither Firefox nor IE liked that and after about 45+ minutes of trying to render the 50k row table the browser would crash.
The (quick and simple) fix was to // the code that displays the report after running it - the CSV files were fine. A better solution would have been to apply the limit to the display code but I didn't need to go there
Just something to bear in mind, hope this helps if you're having any issues as your report grows.
George