Problem:
The Log.log of the Collaboration Manager lists too many users who are authorized to access the site.
Cause:
As soon as a user was ever authorized for the template or was used in a PeoplePicker, this user is listed in the AllUsers List of the web.
Solution:
Remove unnecessary users via Powershell:
$w = $get-spweb [URL]
$au = $w.AllUsers
$au.Remove([claim|domain|user])