We have had general speed issues with 2.3.1 but this seems to be common. Some of our sites having better network connectivity than others compounds this.
However, pluginfile.php loading files seems to be really slow, e.g. at least 2 minutes to get an 800k powerpoint. The files are on the local file system, in a server with 32gb of RAM and 6 cores of CPU. And not that many users on it consistently (max 100 in the 'logged in users' list is about the most I've seen) - however we know other people are trying to use it as we've had to do a lot of password resets (they expire every 3 months with our network password policy - btw we're using LDAP which has been fine previously).
My colleague who is server admin has tried using some caching tools, but even that isn't a straight art - the latest one he tried was making a big improvement on Friday, but turned off over the weekend due to the memory leak problems we've been having, then when that same caching was switched on this morning it ground to a halt!
If anyone has any ideas why this particular component is so troublesome they would be appreciated.