So this is an all or nothing setting,
Well, in a nutshell, yes. When individual user preferences are involved it's not easy (and probably not useful) to cache the whole data.
However, you're already caching the images, yes? Surely that's the bulk of the load for these filters.
If the load continues to be a problem, the site admin can make the decision to force all user preferences for maths display one way or the other ...