User List in a module only assigns space to users who have access to the module

I think it would be a good idea if the user list would only create space for the users who are in a group with access to the module.

In a scenario where a set of super users need to do their own variance analysis that involves lots of calculations, having all of the users who also don't have access to the module will lead to lots of extra space being created that will never be used.

I noticed that an idea had been raised to do a subset of the user list but this had been marked as not planned.