Hi @cmetzcus, thanks for messaging the Community!
The 'Everyone at' group is intentionally meant to store all members of the current Dropbox Business team as an easy way to invite everyone to access folders.
If need be, you can remove the group from folders and instead create smaller groups with only the selected members you wanted.
If you have any further queries, feel free to message back.