The checks and balances needed in these systems are straightforward. They fall out of the required list of elements for a review system to be successful.

1. Require a verified reviewer – Whatever this means in context. Over time we will see OpenID being used to build these systems and identities can be used across systems and potentially aggregated. For now, the walled garden is probably the de facto standard – but that is changing. Systems should definitely not allow anonymous comments to have the same ‘weight’ in whatever scoring or rating system is put in place. Perhaps a side area for anonymous chatter that fades quickly from view.
2. Require a comment/rating – This really does require full text for full explanation and context, everything else is too simple. However, that being said, a simple Lykert scale for a few attributes makes available the generation of statistics and otherwise easier analysis – full text is much harder to count (but much richer).
3. Require a reviewee – The item being reviewed should be a unique entity in the system. For systems where the reviewee is actually a person, and not an event or location or album, the reviewee should be a first class entity as well – able to leave their own feedback and comments/reviews. Here is where the power lies for future development.
4. Make available notifications for all parties involved – This is probably the most important. There should always remain the potential for further conversation about any review/rating. The system should assist in that regard – removing any/all barriers to further discussion and digging to the real facts and circumstances. RSS, email (opt-in or otherwise) should always be made available to all parties; reviewer, reviewee and others.
5. Administrators of the system should always have the right to remove – Probably the most contentious point, but necessary. The administrators should always reserve the right to edit what is said in the system, assuming it’s held in a single place and arguably a full record of what was edited. With a distributed system, this is nigh impossible, but we haven’t seen any like that yet. Distributed systems change the calculus dramatically for all elements of reputation, so this is a fair concession at this point in time.

All systems of this type benefit from the network effects they can generate. The value in the network definitely follows Metcalfe’s Law as the utility of any review area will be significantly greater the greater number of users and reviews posted. It might go without saying, then, that none of the above pieces really matter if you don’t have a proper threshold of activity. The long tail of networks doesn’t win in this regard. Only the big part of the curve gets to play this game today if the goal of the site is global significance (eBay, Amazon, etc.). If a network you are building has a lower bar for success (only your friends, or your school/job), then success can be reached much more easily with less users and activity.

I think that when reputation and identity are involved – these types of systems should definitely not allow for anonymous comments within the walls of the system itself. There is a place for anonymity, almost always, but it doesn’t have to exist formally within the channel. We can always whisper in the hallway or talk around the watercooler – or even print out anonymous rants on the office printer and leave them for co-workers to find and muse over… but the types of effects that recorded/refindable/searchable anonymous comments/rants can have for the person being reviewed are way too dangerous. Like Liz says in a later comment – faculty are much more worried about a boss seeing these potentially slanderous and false comments than they are about potential students seeing them. And without any real recourse, the deck is way too stacked against the reviewee.

A very different question posed later is also a good one. Should these comments be dissolved over time? Do they lose their temporal relevance as time passes and the class and the students and, no doubt, the professor has had a chance to change. If I’m an incoming student, does a review posted 7 years ago really have any relevance to me? Should it? How would I evaluate that?

I’ll write a little more about the pieces of a reputation/review system that I feel are necessary in the next post.

Update: Necessary pieces of a rating and review system