Because the setting is immediately applied (open the Mute List and see how the group is immediately added or removed as you check or uncheck the box). However, for UI coherency (and to actually allow the "Cancel" button to cancel this setting change), I modified the logic to make the setting apply only on press on "Apply" or "OK" buttons.
Yes, and it is also true for muted avatars speaking in groups... These bugs have been fixed for the next release.
Or even simpler, by unchecking the "Include IMs in chat console" setting in the "IM & Logs" tab of the Preferences floater (I didn't see the bug when I tested the code because I always keep this setting off...).