This isn't a topic to discuss elitist philosophies, but rather to discuss the actual usefulness of the rating system. Bare in mind this is only about jj2, and obviously about how to rate creations for jj2.

At the moment, I don't particularly have a solid grasp of the rating system, which means ratings that use decimals such as 8.3, 4.9 and 9.8 are not clearly defined. I do, however, have a solid grasp on ratings such as 8.0, 9.0, 7.0 etc. That is because a level with an 8 rating is a "recommended" upload, thus a quite standard upload should be worthy of the rating.

Here's my current rating system for uploads (just for example):

1: Very Poor - Laughable, Awful, doesn't function "properly" at all for what it sets out to do. A piece of crap literally.
2: Poor - All around bad upload, nothing fuctions "properly", although there are signs of "effort".
3: Well Below Average - Lots of things don't function "properly", but there are definite signs of effort.
4: Below Average - A well established effort, but certain things don't work as they should.
5: Average - A properly usable upload, but lots of things can be enhanced.
6: Above Average - A properly usable upload that has certain "notable" aspects.
7: Well Above Average - Lots of notable aspects, but not quite recommended, and may also contain aspects that have both positive and negative side effects (I consider certain Birdie levels to be within this category ).
8: Good - An outstanding upload, excellent, noteworthy, whole.
9: Great - An often very notable, and/or gigantic upload, something "big", although not everything that's "big" deserves a 9! (I consider stuff like devres to be in this category)
10: Very Great - An utmost, extremely well made, magnificent upload.

I also posted this topic in order to (hopefully) sort out some ideas concerning the rating system. I talked with PurpleJazz recently about a "grading" system that, imo, works much better than the current rating system. I'd rather not explain the grading system, since it isn't my idea to begin with, so I'm hoping PJ will reply to this topic and explain the entire concept.

It's more about being unable to pinpoint .1 differences in the rating to give levels these days. Choosing between 8.2 and 8.3, for example, is a fairly arbitrary decision, and may be determined by one or two very small factors. However, most JCSers still seem to take tiny differences in the rating very seriously, thinking a level that obtained an 8.3 is necessairly better than a level that obtained an 8.2.

To make it even more simple, you can also give a level a rating with an certain amount of stars. like in a range of 1 to 6 stars. But I guess this will sound too simple to most of the people. But I've seen these rating systems a lot before on other sites.

Something like that would be a lot easier to manage.

Something like 8.0 or 8.5 would have very little difference; I suppose an 8.5 would be a "somewhat great upload", but I'm not sure. Ratings such as 8.2-8.3 or 8.7-8.8 might as well be the same for uploads. I don't think you can really pinpoint ratings so accurately for jj2 uploads. It's mostly opinion based anyway.

I actually do have an idea of what kind of upload deserves a 10: It has to be all-versatile for the purpose of the upload. Please consider the logic brought up in this topic. I suppose a "Very Great" CTF level would work exceptionally well in all game sizes, cater exceptionally well to all aspects that constitute CTF such as RTSing, defense etc, and, of course, can not be overcome by "virtue". "Very Great" Tilesets could have something along the lines of "Photorealistic graphics that don't look out of place in JJ2's style". They, of course, have to be all-versatile as well, with things like usability and no room to enhance.

Whichever the case, I didn't take opinions into account, so I'm not saying what I've stated above is absolute. I tend to take a "playground philosophy" approach to level design. Basically, the chosen theme caters to aspects such as the gameplay, eyecandy, music etc, and the players decide if they want to play in that "setting" (I even choose themes that fit specifically into JJ2. Golem Landscapes, for example, fits because it's a fictional creature, and also because JJ2 has been about comically "twisting" popular culture themes).

I do think that the current rating system is needlessly finegrained (there's basically 100 different ratings possible) and making it hard to really attach a common value to a rating. I'm personally in favor of a simpler system, with five stars or something, or even a binary "useful" vs "not useful" rating for, for example, utilities. The big problem is obviously that there's thousands of uploads using the current system and having two separate rating systems on one site isn't a very good idea.

I personally think that the system is fine as it is, it's just too inactive... I have planned to do a temporary fix for that

As for the issue of older reviews with a different rating system: How about simply converting the old ratings to match with the new rating system, like how Youtube did when changing its 5 star rating system to a Y/N system? For example, a rating of 8.2 could be rounded to 4 stars out of 5, a 9.6 could be rounded to 5/5, a 3.0 could be rounded to 1.5/5, etc.

You would still get the .1 differences in average rating, but forcing people to choose between 10 ratings (5 stars with 0.5 differences), would be mean more accurate reviewing.

That's always a possibility of course, though it depends on what new system is chosen (if any)

Yeah, but that's only a solution for download ratings. What's going to happen with reviews that give such ratings though? If you leave them as they are, changing the scale to 5 stars, it's going to look confusing, especially for newcomers who see them and wonder how can they set 4.35 of a star rating themselves. On the other hand, rounding them to the nearest full "star" would affect some of the total ratings. For example Episode 5 would end up with a total rating of exactly 5 stars because none of its reviews sets rating below 9. Uploads rated only by one person as (for example) 7.0 would have to change their rating to either 6.0 or 8.0 (3 or 4 stars), what makes quite a difference. I might be the only one who sees it as an issue though, in that case excuse me, trying to ruin your plan is not my intention.

Besides, I think that 5 possible ratings is already too little. If one wants to say that something is almost perfect, just has a small yet bugging him flaw, the choice between 4 and 5 is going to be a serious dilemma, considering 4 a rating kinda suitable for every decent level, while 5 - the one stating perfection.

Then just give it a 4.

If you give star ratings, half-stars should be possible.

Originally Posted by Jake

If you give star ratings, half-stars should be possible.

Why?

I'm an level maker, I like a bit more exact ratings over a guy having to choose between 3 and 4, and choosing 3 :/

I think half-stars are a good idea too, because 5 levels of detail is somewhat too small a number. 10 would be alright, I think, although that's just me. I do agree that the current system could use a little work.
Oh, and SE, if Episode 5 gets 5 stars, that's for a very good reason.

We already have the Download Recommended/Not Recommended options in place, and average satisfaction, so I imagine that it would be relatively easy to switch to the binary rating. Although that works far better on a large scale. Because level here might only get 1 or 2 ratings, there'd probably be a ton of levels of various quality that have 100% approval with one vote, and something like 5 stars would probably be better to actually make distinctions. Or 10 half-stars.

However, I'm a bit skeptical about how useful any rating system will be. I think no matter what system is in place, most of the problem boils down to people having different opinions about what levels are good, and even about what a given rating means. I've heard a lot of people refusing to give anything a 10 rating because that would imply perfection in their opinion, and no level will be entirely perfect. However, in my mind, the best levels should have the highest rating, so I wouldn't have any qualms giving some of the devres episodes a 10, for example. So with a 5 star system, I think the real difference between a 4 and 5 star level could end up being who reviews the level, not whether the level is entirely perfect or not.

I agree, changing the rating system won't change the people who use it, but I think that a 10-half-star system would be less complex.

Originally Posted by Troglobite

However, I'm a bit skeptical about how useful any rating system will be. I think no matter what system is in place, most of the problem boils down to people having different opinions about what levels are good, and even about what a given rating means. I've heard a lot of people refusing to give anything a 10 rating because that would imply perfection in their opinion, and no level will be entirely perfect. However, in my mind, the best levels should have the highest rating, so I wouldn't have any qualms giving some of the devres episodes a 10, for example. So with a 5 star system, I think the real difference between a 4 and 5 star level could end up being who reviews the level, not whether the level is entirely perfect or not.

Let's think about it like this: You have two tile-sets, and the sets are exactly alike, except one is bigger and has more versatility/flexibility than the other; you can only do more with that set. Which one deserves the higher rating?

I don't really see the issue here. You can choose in your profile what rating system to use -- integers only, integers and .5, integers and .2 and .5 and .7, or full on textbox -- and you are welcome when looking at other people's reviews to round their ratings to the nearest .0 or .5 or whatever you like. You may not believe in the difference between 8.2 and 8.3, and that's fine, but the fact that people use textbox rating in the first place indicates that they prefer the fuller range of options.

Ok then, rating seems to be (entirely?) opinion based, since you can prefer to be more specific(?), or not actually be specific. I'm not saying everyone should rate exactly "right". I don't find much credibility in ratings such as 8.2-8.3; they are unimportant, and they tend to make certain people angry as well.

Levels like E. Prime can get ratings like 1.

As long as every registered member is free to rate, J2O ratings should never be taken seriously. Due to that I personally see no reason to change such a small aspect of the rating system as is the scale, instead of for example creating a "council" which could give downloads a separate, unbiased and allowing comparisons rating. Every download could then have "regular reviews" from normal J2O members and "professional reviews" from council members. Then there would be two total ratings, a "professional rating" being an average of professional reviews, and a "total rating" being an average of all reviews, both regular and professional. This is my opinion on what could really lead to meaningful ratings.

That's an interesting idea, but I don't think there's such a thing as an unbiased review. Well, biased isn't the right word for it, but different people will always have different opinions on how rating systems work. Having thought it through, changing the system probably isn't worth the effort it would take to swap all the reviews. But it's fun to think of some concepts, anyway.
One thing that would be interesting would be to take some older levels and re-review them. Standards have gone way up in the last 10 years or so, and some of the higher rated files on j2o simply aren't as good quality as ones below them. They're just older, from when people's standards were lower. Obviously there are files that deserve the rating just as much as before, but my point stands for a few of the levels, I think.

Originally Posted by FireSworD

Levels like E. Prime can get ratings like 1.

You know, we do monitor the downloads section and read every review to make sure it's legitimate. That could really only happen in a quick review, and those are uniformly meaningless.

Originally Posted by Sir Ementaler

As long as every registered member is free to rate, J2O ratings should never be taken seriously. Due to that I personally see no reason to change such a small aspect of the rating system as is the scale, instead of for example creating a "council" which could give downloads a separate, unbiased and allowing comparisons rating. Every download could then have "regular reviews" from normal J2O members and "professional reviews" from council members. Then there would be two total ratings, a "professional rating" being an average of professional reviews, and a "total rating" being an average of all reviews, both regular and professional. This is my opinion on what could really lead to meaningful ratings.

This was suggested over and over for J2Ov2 and was rightfully rejected. I don't like the idea of an arbitrary split between regular and professional.

My feelings about changing the rating system are the same as Violet's; if we do end up making any changes, however, my suggestion would be to archive the current download section and start anew - considering the amount of uploads like Blade's Battle Pack or 7th Lava Fall, I don't think it would be a bad idea to start from a clean slate, and include uploads only from, say, three years back.

This will probably never happen though...

I don't think changing the rating system would make people rate levels more. The reason I became inactive with rating is because for quite a while the whole site was running slow, and I couldn't take it anymore, also to rethink the way I review levels, but the first reason was the main one. I don't understand that having a .(number) rating can make someone angry, it's just a way of fine-tuning a rating. For some it is not their rating style, but hey everyone has their own ways.

Off-topic: I've seen really ametuer levels with ridiculously high ratings, they tend to happen in quick reviews of their friends. *sigh* I wish friends/members of their clan would actually ask how to improve their levels and such, not just say it's "cool" , when there is obvious signs on how the level can be improved already. Occasionally there's the person they don't know about that gives an honest rating. Beginner level makers that no one knows about tend to get really low ratings. It just shows that bias can seriously blind people. Although it may just be human nature at work.

I don't know if the ratings are the problems with reviews here. I think reviews are not accurate enough to the upload and needs to be expanded more.

Sure, the quick reviews really helped to bring back up the scene of reviews. But the downside is that i sometimes miss reviews that get deleted. The downside of quick reviews is that people don't have to explain much at all. Saying

Cool

or

Nice

is actually enough for the review to stay. Of course i know that the reviews count for 25%. But what's the point of reviewing here..

I don't think it's just the quick reviews fault either. But some reviewers just don't know what they are talking about. There is much to tell about a level, not just

there is a bouncer power up on the left side, and a seeker powerup on the right side which is nice.

The worst: Reviewers that just tell people what you can find in the level and give just a little opinion. (aka repeating the description)

Bit better: Reviewers that tell people what you can find in the level, give their opinion about it and can observe a bit of gameplay. (like considering 2 power ups together is not a good idea).

How it should be: Reviewers that can tell what you can find in the level, give their opinion on that, give arguments for opinions. Can tell what they think should improve, how to improve it, can observe gameplay, know what the level is meant to be for. (theme) Can see what the author wanted to achieve.

Now I personally think that observing gameplay is also hard for someone who is very unskilled in the game himself. I don't have much problems with how the review is set up nicely or crap english, honestly. But it's often tiresome how people show so much confidence in talking nonsense aswell.

Now you probably ask, What does this all have to do with reviews? I think that we have to face that jj2 is a small commnunity and that amount of good reviewers is even smaller. In the past there were more reviews, but not much of them look accurate and helpful to me. And having the quick reviews, lending out an easy possibility to leave behind a short "review" or comment maybe supports the activity, but not the activity of helpful and good reviews.

I like the stars idea so far and I think I would go for the half-star rating system personally. I like the idea SE mentioned aswell, but maybe that is meant to remove unhelpful and senseless reviews. But then again, who decides a review is helpful or not. Perhaps having the review rated 0 out of 5 should stop counting. But a good start would be erasing quick reviews that have no content at all, even if they are quick they should have, otherwise reviewing has no purpose.

Reviewing needs to stay reviewing, no matter how low the activity gets.

Also, another brief thought:
I know that reviewing all depends on that person's opinion, but perhaps the scores could be calculated partially depending on the reviewer's average helpfulness and/or points. I mean, since the points/helpfulness represent how the community likes their reviews, then perhaps their reviews could factor more/less into a level's final score. Not by a huge amount, but just a little bit. Then again, if the points already do something like this, feel free to correct me, as I couldn't find anything saying so.

Originally Posted by FawFul

Reviewing needs to stay reviewing, no matter how low the activity gets.

This is the worst idea in this thread yet

Originally Posted by Obi1mcd

Also, another brief thought:
I know that reviewing all depends on that person's opinion, but perhaps the scores could be calculated partially depending on the reviewer's average helpfulness and/or points. I mean, since the points/helpfulness represent how the community likes their reviews, then perhaps their reviews could factor more/less into a level's final score. Not by a huge amount, but just a little bit. Then again, if the points already do something like this, feel free to correct me, as I couldn't find anything saying so.

I think this makes sense. I would support adding this feature, if it doesn't already exist.

I think part of the reason that's not currently done is that it would mean ratings would be recalculated a lot more frequently. Either every time you viewed a download (e.g. looking at the list of the 30 most recent downloads) its rating would have to be recalculated in case the helpfulness of any of the reviewers had changed, or else every time a user's helpfulness changed, the rating of every download they had ever reviewed would have to be recalculated, which in some cases number in the hundreds.

How about making the rating based on the 'rate value' of the reviewer at the moment he makes the review? That would save you from recalculating.

Sounds like easy to abuse by 100% helpfulness 1 out of 1 vote accidental newcomers. Edit: I don't want to sound like complaining too much, I like the idea of basing rating weight on helpfulness and this is also one of the improvements I considered. Either what Violet said or what I just said would most probably happen though.

It would obviously need some though and tweaking. I imagine you could create a system where people by default have 50% helpfulness until they get 20 votes or something. And using the 'rate value' at the moment of the review being written sounds reasonable to me. Especially because I could imagine someone's really old review might not be as high quality as ones after they've been playing Jazz for longer and gotten a better feel for what makes a good review.

Originally Posted by Troglobite

It would obviously need some though and tweaking. I imagine you could create a system where people by default have 50% helpfulness until they get 20 votes or something. And using the 'rate value' at the moment of the review being written sounds reasonable to me. Especially because I could imagine someone's really old review might not be as high quality as ones after they've been playing Jazz for longer and gotten a better feel for what makes a good review.

The problem I'm seeing with all of this is that not only reviews are based on opinions. There's also opinions towards the review and the reviewee. And honestly I could see someone thinking "oh, HE wrote a review. HE's an idiot, I'm going to weight my opinions fully against his review and downvote him."

Even if that person made a decent review, their opinion becomes worthless. Many people could have different opinions towards a review, but I've observed that people within this community would have a tendency to more quickly mark down a review than to mark it up.

Also when you have a rating system based on a person's input, it puts pressure on that person which could cause them to withhold their opinions in fear of being harshly judged negatively. It's already bad enough to be posting your opinions on the internet, accessible to the world, but by having people rate them, it causes further hesitation. And honestly I'd rather read somebody's non-judged opinions; whether it differs or not from my own makes no difference but having vast inputs would make for the best final rating. We don't want other people's valid inputs withheld and we certainly don't want to keep people from conducting a review simply because they disagree with the majority. That would be counterproductive to the whole idea of "fixing" the review system. The review system already isn't used as heavily as it should be. Why deter people further?

As an alternative to a user-based rating system, why not a review-based rating system? If a review is, in your opinion, invalid, click the thumbs down button. If it is, click the thumbs up button. But don't, of course, keep a count on the user's profile of how many thumbs down or thumbs up they've received because that just deters people to follow a higher-rated person's opinions so they get higher ratings while a person on the lower end continues to drop. Also keep in mind that there are juvenile reviews from the past. People grow up, learn more, and change. Those reviews may not mean anything to the reviewer's overall present-day review quality, so would it be fair to use that against them? (I have no opinion on that matter, you decide)

I also second Unknown Rabbit's response. That would create a mess... And initially, ratings will definitely be bouncing all over this place if this system were to be enacted. Its just an overall bad idea. A review should be based on the material included in the review, there are already filters enacted and used and I'm sure they would continue to be used; user-based rating systems just aren't a good form of quality control.

Originally Posted by SilverBolt

"oh, HE wrote a review. HE's an idiot, I'm going to weight my opinions fully against his review and downvote him."

Download admins should remove those.

Originally Posted by SilverBolt

Also when you have a rating system based on a person's input, it puts pressure on that person which could cause them to withhold their opinions in fear of being harshly judged negatively.

Again, admins.

Originally Posted by SilverBolt

As an alternative to a user-based rating system, why not a review-based rating system? If a review is, in your opinion, invalid, click the thumbs down button. If it is, click the thumbs up button.

It took me a while to understand what you mean. I'd prefer the global rating system.

Originally Posted by SilverBolt

Those reviews may not mean anything to the reviewer's overall present-day review quality, so would it be fair to use that against them? (I have no opinion on that matter, you decide)

Yes, this is the problem with the system. Maybe the system should be based on reviews that are newer rather than every review being the same value?

Originally Posted by SilverBolt

I also second Unknown Rabbit's response. That would create a mess... And initially, ratings will definitely be bouncing all over this place if this system were to be enacted.

I think part of the reason that's not currently done is that it would mean ratings would be recalculated a lot more frequently.

How about making the rating based on the 'rate value' of the reviewer at the moment he makes the review? That would save you from recalculating.

And one more thing; There should be a forced comment about the review, if an user clicks the helpful/not button. This would help admins decide between a proper comment and an hateful negrep.

I supposed SilverBolt makes a good point. Even reviews of reviews are based on opinions. However, I don't think that this necessarily makes it a useless system. Over time, after someone makes several good reviews, people will hopefully change their opinion of said person, and read through before downvoting. So yes, it could make some reviews less valuable than others. But in the long run, it should on average, make the reviews and scores more accurate. Also, I think that we could use the "rate value" to weight the reviewer's score without simply multiplying rate value times the score they give. Maybe anyone's review would always have some base value, then well rated reviewers scores would be given a bit of extra sway. This means that even someone with a 0% rate value would still be able to have some effect on the score, so they wouldn't feel that giving reviews was pointless.

Our current system already has a review voting system that keeps track of a permanent "Average Helpfulness", and I can see how many votes were involved in creating that score. And I don't think that this has discouraged reviews.

A review based rating system might work, but I think that again, our community is a bit small. Most review probably will never get more than a couple votes, and this leads to a really small sample size, so I think that there would be too many short pointless reviews that get one thumbs up and have too much sway, or long, detailed reviews that one person disagrees with and end up counting for less than they should.

Jake's suggestions might help, but I think that even with forcing comments, most people would probably say, "Nice." or "I disagree," and it would be impossible for admins to tell the real votes apart from biased votes for friends or hateful downvotes. Not to mention we're asking the admins to micromanage a rating system of a rating system.

That't the point of the comments on ratings: If there isn't a proper explanation, the rating is invalid and must be deleted. 'Nice' 'I disagree' are these.

And then let people rate those too, and then add a reputation system for ratings for comments for reviews for downloads so it's all fair. I mean, it's getting too complex, who would want to write long in-depth comments on reviews already? And who would want to moderate them?

I can (moderate).

Originally Posted by Sir Ementaler

And then let people rate those too, and then add a reputation system for ratings for comments for reviews for downloads so it's all fair. I mean, it's getting too complex, who would want to write long in-depth comments on reviews already? And who would want to moderate them?

Agreed. If we don't have many people who want to put in enough effort to write a review for something interesting like a level, then I don't think we'd get nearly enough people to review reviews.

Originally Posted by Troglobite

Over time, after someone makes several good reviews, people will hopefully change their opinion of said person, and read through before downvoting.

Unfortunately, that's not how it generally works. Usually when someone's rep is at a low, nobody even bothers to listen to them, despite whether their quality has improved or not. Take Crazy Rabbit or cooldude as an example. Doesn't matter how they behave, once they've joined your server, you've already sighed because of past experiences with them. This community isn't exactly full of free spirits from what I've noticed.

Originally Posted by Troglobite

In the long run, it should on average, make the reviews and scores more accurate. Also, I think that we could use the "rate value" to weight the reviewer's score without simply multiplying rate value times the score they give. Maybe anyone's review would always have some base value, then well rated reviewers scores would be given a bit of extra sway. This means that even someone with a 0% rate value would still be able to have some effect on the score, so they wouldn't feel that giving reviews was pointless.

Excellent idea! Except... unfortunately, this system is attempting to create quality control. But this goes back to my previous idea of just rating the review itself and not the reviewer. A system that rates the reviewer would be along the same lines as the current rating system which seems has caused constant bickering among members. Even as it is, I hate posting these responses. You might not like my contradictions and neg-rep me... Or God forbid, Unknown Rabbit may disagree and do that. Once I'm in the negatives, how long would you foresee it taking before I got back into the positives? And how valid would any of my statements be? A user-based rating system just doesn't seem to promote creation, if anything, it destroys it.

Originally Posted by Troglobite

Our current system already has a review voting system that keeps track of a permanent "Average Helpfulness", and I can see how many votes were involved in creating that score. And I don't think that this has discouraged reviews.

You ought to re-think that one... I personally don't create any reviews for that reason, I'm not good at it and I don't want to be judged based on them. I would register and review if I found a level deserving of my time. But by enacting this system, I know for a fact that it would further my hesitation on attempting to write a review because this review would have to be perfect, otherwise I'm seen as an idiot. The whole reason our review system needs to be "fixed" is because its not used as heavily as it should be and apparently we need forms of quality control.

Originally Posted by Troglobite

A review based rating system might work, but I think that again, our community is a bit small. Most review probably will never get more than a couple votes, and this leads to a really small sample size, so I think that there would be too many short pointless reviews that get one thumbs up and have too much sway, or long, detailed reviews that one person disagrees with and end up counting for less than they should.

This community is small, indeed. And pretty tightly-knit. That's more of a downfall, since you won't have as many different opinions on a single review. We all know each other and are more bias towards those we like as opposed to those we don't like. In a larger community, this system would make a bit more sense since you wouldn't know, nor have the time to bother looking into a reviewer's past record or holding it against them. In a community like this, however, the famed become more famous while the less famous become slaves to the system.

Originally Posted by Troglobite

Jake's suggestions might help, but I think that even with forcing comments, most people would probably say, "Nice." or "I disagree," and it would be impossible for admins to tell the real votes apart from biased votes for friends or hateful downvotes. Not to mention we're asking the admins to micromanage a rating system of a rating system.

I completely agree with this which is exactly why I state that a user-based rating system does not seem fair, in my opinion. As for comments on reviews, you might as well leave that to be a free market. I don't even see that much of a problem with our current system...

Originally Posted by Jake

That't the point of the comments on ratings: If there isn't a proper explanation, the rating is invalid and must be deleted. 'Nice' 'I disagree' are these.

Or you could report it. But really, I'm ok with comments, but "reputation-based" is a really bad idea in my honest opinion. And comments should be regulated in the same manner that posts on this forum are (in the sense that if they're even remotely on-topic, they're fair).

Originally Posted by Sir Ementaler

And then let people rate those too, and then add a reputation system for ratings for comments for reviews for downloads so it's all fair. I mean, it's getting too complex, who would want to write long in-depth comments on reviews already? And who would want to moderate them?

Originally Posted by SilverBolt

and apparently we need forms of quality control.