Hey @Thore
as per Retool's documentation
The All users group automatically includes all current organization users. You cannot modify group membership directly, you must invite or delete organization users. You can, however, configure access rules for the All users group to set the default level of access for new users.
You can however remove all accesses for All Users:
So, all users will always include all people in your account, but that doesn't need to impact all of your other groups' permissions.