Right, custom.css seems a logical place to put things, but if you think about it, your web server never normally writes to files in the /moodle directory except during installs & upgrades. If it needs to change files on the fly, it needs to use /moodledata. For example, custom language strings, custom sql reports, etc., go there.
The custom.css file is just the standard name of the main css file for a theme: even Clean and More have that file in the style directory. That file is created by the theme developer and will never changed unless you upgrade the theme.
I would suppose that it is called custom because it is a customization of base, Bootstrapbase.
If you make a dependent theme, you will always have your own /style directory with a custom.css file in it to control your theme's customizations which override styles from base. Look at the files for Clean or More for examples.
What a user puts in the custom CSS box for a theme via the admin interface is stored in the database not in that file.
Now, things are different for Boost and themes which are now using scss. I don't know much about them at all but the structures are quite different which makes sense since scss can @import things like crazy and is a different way of working.