Jul 21, 2011, 04:35 PM | |
My thoughts on the rating system for J2O downloads
This isn't a topic to discuss elitist philosophies, but rather to discuss the actual usefulness of the rating system. Bare in mind this is only about jj2, and obviously about how to rate creations for jj2.
At the moment, I don't particularly have a solid grasp of the rating system, which means ratings that use decimals such as 8.3, 4.9 and 9.8 are not clearly defined. I do, however, have a solid grasp on ratings such as 8.0, 9.0, 7.0 etc. That is because a level with an 8 rating is a "recommended" upload, thus a quite standard upload should be worthy of the rating. Here's my current rating system for uploads (just for example): 1: Very Poor - Laughable, Awful, doesn't function "properly" at all for what it sets out to do. A piece of crap literally. 2: Poor - All around bad upload, nothing fuctions "properly", although there are signs of "effort". 3: Well Below Average - Lots of things don't function "properly", but there are definite signs of effort. 4: Below Average - A well established effort, but certain things don't work as they should. 5: Average - A properly usable upload, but lots of things can be enhanced. 6: Above Average - A properly usable upload that has certain "notable" aspects. 7: Well Above Average - Lots of notable aspects, but not quite recommended, and may also contain aspects that have both positive and negative side effects (I consider certain Birdie levels to be within this category ![]() 8: Good - An outstanding upload, excellent, noteworthy, whole. 9: Great - An often very notable, and/or gigantic upload, something "big", although not everything that's "big" deserves a 9! (I consider stuff like devres to be in this category) 10: Very Great - An utmost, extremely well made, magnificent upload. I also posted this topic in order to (hopefully) sort out some ideas concerning the rating system. I talked with PurpleJazz recently about a "grading" system that, imo, works much better than the current rating system. I'd rather not explain the grading system, since it isn't my idea to begin with, so I'm hoping PJ will reply to this topic and explain the entire concept. Last edited by FireSworD; Jun 15, 2016 at 02:40 AM. |
Jul 22, 2011, 01:12 AM | |
It's more about being unable to pinpoint .1 differences in the rating to give levels these days. Choosing between 8.2 and 8.3, for example, is a fairly arbitrary decision, and may be determined by one or two very small factors. However, most JCSers still seem to take tiny differences in the rating very seriously, thinking a level that obtained an 8.3 is necessairly better than a level that obtained an 8.2.
|
Jul 22, 2011, 02:38 PM | |
To make it even more simple, you can also give a level a rating with an certain amount of stars. like in a range of 1 to 6 stars. But I guess this will sound too simple to most of the people. But I've seen these rating systems a lot before on other sites.
__________________
|
Jul 22, 2011, 03:19 PM | |
Something like that would be a lot easier to manage.
Something like 8.0 or 8.5 would have very little difference; I suppose an 8.5 would be a "somewhat great upload", but I'm not sure. Ratings such as 8.2-8.3 or 8.7-8.8 might as well be the same for uploads. I don't think you can really pinpoint ratings so accurately for jj2 uploads. It's mostly opinion based anyway. I actually do have an idea of what kind of upload deserves a 10: It has to be all-versatile for the purpose of the upload. Please consider the logic brought up in this topic. I suppose a "Very Great" CTF level would work exceptionally well in all game sizes, cater exceptionally well to all aspects that constitute CTF such as RTSing, defense etc, and, of course, can not be overcome by "virtue". "Very Great" Tilesets could have something along the lines of "Photorealistic graphics that don't look out of place in JJ2's style". They, of course, have to be all-versatile as well, with things like usability and no room to enhance. Whichever the case, I didn't take opinions into account, so I'm not saying what I've stated above is absolute. I tend to take a "playground philosophy" approach to level design. Basically, the chosen theme caters to aspects such as the gameplay, eyecandy, music etc, and the players decide if they want to play in that "setting" (I even choose themes that fit specifically into JJ2. Golem Landscapes, for example, fits because it's a fictional creature, and also because JJ2 has been about comically "twisting" popular culture themes). Last edited by FireSworD; Feb 22, 2017 at 11:55 PM. |
Jul 23, 2011, 03:19 AM | |
I do think that the current rating system is needlessly finegrained (there's basically 100 different ratings possible) and making it hard to really attach a common value to a rating. I'm personally in favor of a simpler system, with five stars or something, or even a binary "useful" vs "not useful" rating for, for example, utilities. The big problem is obviously that there's thousands of uploads using the current system and having two separate rating systems on one site isn't a very good idea.
|
Jul 23, 2011, 03:38 AM | |
As for the issue of older reviews with a different rating system: How about simply converting the old ratings to match with the new rating system, like how Youtube did when changing its 5 star rating system to a Y/N system? For example, a rating of 8.2 could be rounded to 4 stars out of 5, a 9.6 could be rounded to 5/5, a 3.0 could be rounded to 1.5/5, etc.
You would still get the .1 differences in average rating, but forcing people to choose between 10 ratings (5 stars with 0.5 differences), would be mean more accurate reviewing. |
Jul 23, 2011, 04:45 AM | |
Yeah, but that's only a solution for download ratings. What's going to happen with reviews that give such ratings though? If you leave them as they are, changing the scale to 5 stars, it's going to look confusing, especially for newcomers who see them and wonder how can they set 4.35 of a star rating themselves. On the other hand, rounding them to the nearest full "star" would affect some of the total ratings. For example Episode 5 would end up with a total rating of exactly 5 stars because none of its reviews sets rating below 9. Uploads rated only by one person as (for example) 7.0 would have to change their rating to either 6.0 or 8.0 (3 or 4 stars), what makes quite a difference. I might be the only one who sees it as an issue though, in that case excuse me, trying to ruin your plan is not my intention.
Besides, I think that 5 possible ratings is already too little. If one wants to say that something is almost perfect, just has a small yet bugging him flaw, the choice between 4 and 5 is going to be a serious dilemma, considering 4 a rating kinda suitable for every decent level, while 5 - the one stating perfection.
__________________
I am an official JJ2+ programmer and this has been an official JJ2+ statement. |
Jul 23, 2011, 06:39 AM | |
I think half-stars are a good idea too, because 5 levels of detail is somewhat too small a number. 10 would be alright, I think, although that's just me. I do agree that the current system could use a little work.
Oh, and SE, if Episode 5 gets 5 stars, that's for a very good reason. ![]()
__________________
Define 'normal'. |
Jul 23, 2011, 08:52 AM | |
We already have the Download Recommended/Not Recommended options in place, and average satisfaction, so I imagine that it would be relatively easy to switch to the binary rating. Although that works far better on a large scale. Because level here might only get 1 or 2 ratings, there'd probably be a ton of levels of various quality that have 100% approval with one vote, and something like 5 stars would probably be better to actually make distinctions. Or 10 half-stars.
However, I'm a bit skeptical about how useful any rating system will be. I think no matter what system is in place, most of the problem boils down to people having different opinions about what levels are good, and even about what a given rating means. I've heard a lot of people refusing to give anything a 10 rating because that would imply perfection in their opinion, and no level will be entirely perfect. However, in my mind, the best levels should have the highest rating, so I wouldn't have any qualms giving some of the devres episodes a 10, for example. So with a 5 star system, I think the real difference between a 4 and 5 star level could end up being who reviews the level, not whether the level is entirely perfect or not.
__________________
![]() Lexicographer: Someone who writes dictionaries Neophyte: A novice, or newbie Hemisemidemiquaver: In music, a sixty-fourth note Exit Troglobite, Stage Left |
Jul 23, 2011, 09:23 PM | ||
Quote:
Last edited by FireSworD; Jun 14, 2016 at 12:24 AM. |
Jul 24, 2011, 12:39 AM | |
I don't really see the issue here. You can choose in your profile what rating system to use -- integers only, integers and .5, integers and .2 and .5 and .7, or full on textbox -- and you are welcome when looking at other people's reviews to round their ratings to the nearest .0 or .5 or whatever you like. You may not believe in the difference between 8.2 and 8.3, and that's fine, but the fact that people use textbox rating in the first place indicates that they prefer the fuller range of options.
|
Jul 24, 2011, 02:42 AM | |
Ok then, rating seems to be (entirely?) opinion based, since you can prefer to be more specific(?), or not actually be specific. I'm not saying everyone should rate exactly "right". I don't find much credibility in ratings such as 8.2-8.3; they are unimportant, and they tend to make certain people angry as well.
Levels like E. Prime can get ratings like 1. ![]() Last edited by FireSworD; Jun 14, 2016 at 12:26 AM. |
Jul 24, 2011, 08:50 AM | |
As long as every registered member is free to rate, J2O ratings should never be taken seriously. Due to that I personally see no reason to change such a small aspect of the rating system as is the scale, instead of for example creating a "council" which could give downloads a separate, unbiased and allowing comparisons rating. Every download could then have "regular reviews" from normal J2O members and "professional reviews" from council members. Then there would be two total ratings, a "professional rating" being an average of professional reviews, and a "total rating" being an average of all reviews, both regular and professional. This is my opinion on what could really lead to meaningful ratings.
__________________
I am an official JJ2+ programmer and this has been an official JJ2+ statement. |
Jul 24, 2011, 08:58 AM | |
That's an interesting idea, but I don't think there's such a thing as an unbiased review. Well, biased isn't the right word for it, but different people will always have different opinions on how rating systems work. Having thought it through, changing the system probably isn't worth the effort it would take to swap all the reviews. But it's fun to think of some concepts, anyway.
One thing that would be interesting would be to take some older levels and re-review them. Standards have gone way up in the last 10 years or so, and some of the higher rated files on j2o simply aren't as good quality as ones below them. They're just older, from when people's standards were lower. Obviously there are files that deserve the rating just as much as before, but my point stands for a few of the levels, I think.
__________________
Define 'normal'. |
Jul 24, 2011, 10:12 AM | |
You know, we do monitor the downloads section and read every review to make sure it's legitimate. That could really only happen in a quick review, and those are uniformly meaningless.
|
Jul 24, 2011, 10:38 AM | ||
Quote:
My feelings about changing the rating system are the same as Violet's; if we do end up making any changes, however, my suggestion would be to archive the current download section and start anew - considering the amount of uploads like Blade's Battle Pack or 7th Lava Fall, I don't think it would be a bad idea to start from a clean slate, and include uploads only from, say, three years back. This will probably never happen though...
__________________
Mystic Legends http://www.mysticlegends.org/ The Price of Admission - Hoarfrost Hollow - Sacrosanct - other - stuff |
Jul 25, 2011, 12:54 PM | |
I don't think changing the rating system would make people rate levels more. The reason I became inactive with rating is because for quite a while the whole site was running slow, and I couldn't take it anymore, also to rethink the way I review levels, but the first reason was the main one. I don't understand that having a .(number) rating can make someone angry, it's just a way of fine-tuning a rating. For some it is not their rating style, but hey everyone has their own ways.
Off-topic: I've seen really ametuer levels with ridiculously high ratings, they tend to happen in quick reviews of their friends. *sigh* I wish friends/members of their clan would actually ask how to improve their levels and such, not just say it's "cool" , when there is obvious signs on how the level can be improved already. Occasionally there's the person they don't know about that gives an honest rating. Beginner level makers that no one knows about tend to get really low ratings. It just shows that bias can seriously blind people. Although it may just be human nature at work. |
Jul 27, 2011, 06:28 AM | ||||
I don't know if the ratings are the problems with reviews here. I think reviews are not accurate enough to the upload and needs to be expanded more.
Sure, the quick reviews really helped to bring back up the scene of reviews. But the downside is that i sometimes miss reviews that get deleted. The downside of quick reviews is that people don't have to explain much at all. Saying Quote:
Quote:
I don't think it's just the quick reviews fault either. But some reviewers just don't know what they are talking about. There is much to tell about a level, not just Quote:
Bit better: Reviewers that tell people what you can find in the level, give their opinion about it and can observe a bit of gameplay. (like considering 2 power ups together is not a good idea). How it should be: Reviewers that can tell what you can find in the level, give their opinion on that, give arguments for opinions. Can tell what they think should improve, how to improve it, can observe gameplay, know what the level is meant to be for. (theme) Can see what the author wanted to achieve. Now I personally think that observing gameplay is also hard for someone who is very unskilled in the game himself. I don't have much problems with how the review is set up nicely or crap english, honestly. But it's often tiresome how people show so much confidence in talking nonsense aswell. Now you probably ask, What does this all have to do with reviews? I think that we have to face that jj2 is a small commnunity and that amount of good reviewers is even smaller. In the past there were more reviews, but not much of them look accurate and helpful to me. And having the quick reviews, lending out an easy possibility to leave behind a short "review" or comment maybe supports the activity, but not the activity of helpful and good reviews. I like the stars idea so far and I think I would go for the half-star rating system personally. I like the idea SE mentioned aswell, but maybe that is meant to remove unhelpful and senseless reviews. But then again, who decides a review is helpful or not. Perhaps having the review rated 0 out of 5 should stop counting. But a good start would be erasing quick reviews that have no content at all, even if they are quick they should have, otherwise reviewing has no purpose. Reviewing needs to stay reviewing, no matter how low the activity gets.
__________________
|
Jul 27, 2011, 07:04 AM | |
Also, another brief thought:
I know that reviewing all depends on that person's opinion, but perhaps the scores could be calculated partially depending on the reviewer's average helpfulness and/or points. I mean, since the points/helpfulness represent how the community likes their reviews, then perhaps their reviews could factor more/less into a level's final score. Not by a huge amount, but just a little bit. Then again, if the points already do something like this, feel free to correct me, as I couldn't find anything saying so.
__________________
Define 'normal'. |
Jul 27, 2011, 11:17 AM | ||
Quote:
__________________
![]() Lexicographer: Someone who writes dictionaries Neophyte: A novice, or newbie Hemisemidemiquaver: In music, a sixty-fourth note Exit Troglobite, Stage Left |
Jul 27, 2011, 11:30 AM | |
I think part of the reason that's not currently done is that it would mean ratings would be recalculated a lot more frequently. Either every time you viewed a download (e.g. looking at the list of the 30 most recent downloads) its rating would have to be recalculated in case the helpfulness of any of the reviewers had changed, or else every time a user's helpfulness changed, the rating of every download they had ever reviewed would have to be recalculated, which in some cases number in the hundreds.
|
Jul 27, 2011, 01:32 PM | |
Sounds like easy to abuse by 100% helpfulness 1 out of 1 vote accidental newcomers. Edit: I don't want to sound like complaining too much, I like the idea of basing rating weight on helpfulness and this is also one of the improvements I considered. Either what Violet said or what I just said would most probably happen though.
__________________
I am an official JJ2+ programmer and this has been an official JJ2+ statement. |
Jul 27, 2011, 07:08 PM | |
It would obviously need some though and tweaking. I imagine you could create a system where people by default have 50% helpfulness until they get 20 votes or something. And using the 'rate value' at the moment of the review being written sounds reasonable to me. Especially because I could imagine someone's really old review might not be as high quality as ones after they've been playing Jazz for longer and gotten a better feel for what makes a good review.
__________________
![]() Lexicographer: Someone who writes dictionaries Neophyte: A novice, or newbie Hemisemidemiquaver: In music, a sixty-fourth note Exit Troglobite, Stage Left |
Jul 28, 2011, 10:50 PM | ||
Quote:
Even if that person made a decent review, their opinion becomes worthless. Many people could have different opinions towards a review, but I've observed that people within this community would have a tendency to more quickly mark down a review than to mark it up. Also when you have a rating system based on a person's input, it puts pressure on that person which could cause them to withhold their opinions As an alternative to a user-based rating system, why not a review-based rating system? If a review is, in your opinion, invalid, click the thumbs down button. If it is, click the thumbs up button. But don't, of course, keep a count on the user's profile of how many thumbs down or thumbs up they've received because that just deters people to follow a higher-rated person's opinions so they get higher ratings while a person on the lower end continues to drop. Also keep in mind that there are juvenile reviews from the past. People grow up, learn more, and change. Those reviews may not mean anything to the reviewer's overall present-day review quality, so would it be fair to use that against them? (I have no opinion on that matter, you decide) I also second Unknown Rabbit's response. That would create a mess... And initially, ratings will definitely be bouncing all over this place if this system were to be enacted. Its just an overall bad idea. A review should be based on the material included in the review, there are already filters enacted and used and I'm sure they would continue to be used; user-based rating systems just aren't a good form of quality control.
__________________
is recharging [██▒▒▒▒▒▒] 28% ![]() ![]() |
Jul 29, 2011, 03:57 AM | ||||||||
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
|
Jul 29, 2011, 09:25 AM | |
I supposed SilverBolt makes a good point. Even reviews of reviews are based on opinions. However, I don't think that this necessarily makes it a useless system. Over time, after someone makes several good reviews, people will hopefully change their opinion of said person, and read through before downvoting. So yes, it could make some reviews less valuable than others. But in the long run, it should on average, make the reviews and scores more accurate. Also, I think that we could use the "rate value" to weight the reviewer's score without simply multiplying rate value times the score they give. Maybe anyone's review would always have some base value, then well rated reviewers scores would be given a bit of extra sway. This means that even someone with a 0% rate value would still be able to have some effect on the score, so they wouldn't feel that giving reviews was pointless.
Our current system already has a review voting system that keeps track of a permanent "Average Helpfulness", and I can see how many votes were involved in creating that score. And I don't think that this has discouraged reviews. A review based rating system might work, but I think that again, our community is a bit small. Most review probably will never get more than a couple votes, and this leads to a really small sample size, so I think that there would be too many short pointless reviews that get one thumbs up and have too much sway, or long, detailed reviews that one person disagrees with and end up counting for less than they should. Jake's suggestions might help, but I think that even with forcing comments, most people would probably say, "Nice." or "I disagree," and it would be impossible for admins to tell the real votes apart from biased votes for friends or hateful downvotes. Not to mention we're asking the admins to micromanage a rating system of a rating system.
__________________
![]() Lexicographer: Someone who writes dictionaries Neophyte: A novice, or newbie Hemisemidemiquaver: In music, a sixty-fourth note Exit Troglobite, Stage Left |
Jul 29, 2011, 11:47 AM | |
And then let people rate those too, and then add a reputation system for ratings for comments for reviews for downloads so it's all fair. I mean, it's getting too complex, who would want to write long in-depth comments on reviews already? And who would want to moderate them?
__________________
I am an official JJ2+ programmer and this has been an official JJ2+ statement. |
Jul 29, 2011, 02:23 PM | ||
Quote:
__________________
![]() Lexicographer: Someone who writes dictionaries Neophyte: A novice, or newbie Hemisemidemiquaver: In music, a sixty-fourth note Exit Troglobite, Stage Left |
Jul 29, 2011, 08:52 PM | ||||||||
Quote:
Quote:
Quote:
![]() Quote:
Quote:
I completely agree with this which is exactly why I state that a user-based rating system does not seem fair, in my opinion. As for comments on reviews, you might as well leave that to be a free market. I don't even see that much of a problem with our current system... Quote:
Quote:
__________________
is recharging [██▒▒▒▒▒▒] 28% ![]() ![]() |
![]() |
«
Previous Thread
|
Next Thread
»
Thread Tools | |
|
|
All times are GMT -8. The time now is 11:04 AM.
Jazz2Online © 1999-INFINITY (Site Credits). Jazz Jackrabbit, Jazz Jackrabbit 2, Jazz Jackrabbit Advance and all related trademarks and media are ™ and © Epic Games. Lori Jackrabbit is © Dean Dodrill. J2O development powered by Loops of Fury and Chemical Beats. Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Original site design by Ovi Demetrian. DrJones is the puppet master. Eat your lima beans, Johnny.