Unless the non-idle spectators are talkative, then there isn't really any way for the game to know who is idle and who isn't. If they aren't giving any input then as far as the server is concerned they'd be idle. The only thing you could do is kick anyone who hasn't said anything in the past X minutes, and that wouldn't be much fun for people who are trying to watch.