There's a function called resolve_theme in lib/pagelib.php which decides which theme to use based on all the various theme settings, and I can edit that to e.g. return a different theme based on the URL or whatever. But I can't override that from a theme, you need to edit core Moodle.
On the other hand I can override standard_head_html in a renderer, which means I can point to any CSS files I like and so could point to a different theme as long as it was installed. However, that "external" theme can influence the the layout file that is used, it has to use one from the current theme.
So I think the best I can do is to include a different theme's CSS based on whatever variables I want but require that the relevant admin.php layout from that theme is included in order for it to work.
It's good enough to experiment with and see if there's an easy way to divide Moodle into backstage admin settings and public theme'd sections and provide two seperate themes for each. If it works out then maybe a UI could be added and the change integrated into resolve_theme.
So any other suggestions for what divides backstage moodle from public facing moodle? I'll try having "admin" in the URL and having a layout of "admin", but I get the feeling it might devolve into a hand-picked list.