The All users group automatically includes all current organization users. You cannot modify group membership directly, you must invite or delete organization users. You can, however, configure access rules for the All users group to set the default level of access for new users.
You can however remove all accesses for All Users: