Hm - are you talking about performance of file serving requests i.e. if the user requests a large number of supplementary files (images, JS, CSS etc) all at once?
I think that's a fairly specialist issue - and the system makes a number of database requests as part of a pluginfile.php request, the majority of which are not to the files table. In fact the files table queries are usually rather simple. So basically, I'm not sure it is the first place to look for performance improvement in fileserving. There's probably still room for more generic improvements to the access-checking/setup.php type code.
Regarding file API and performance, what I'd personally like is an allowed way to access the real path of the file - if available - for read-only use. This would save unnecessary copies of files to temp directory for various processing in various custom modules/tasks I've written, which can take significant time and I/O in the case of large files. In other words, if there was a
$path = $file->allocate_read_only_path()
So if your file is stored in the local filesystem it just returns the path and does nothing, if it's stored elsewhere then the allocate/release functions would do the same as the current function that saves it to a temp path, then delete it on release.