maybe it would be better to post a thread clearly named POLE or such.

That would not be clear at all. We don't really need a thread about poles. Aren't pedants irritating?

I'm not convinced either of these polls will garner many votes, but I'll at least try to get the ball rolling.

Vote: Categories

God, just waiting for a typo huh?

I reserve my vote for later. I am undecided tbh.

Vote: Categories

what are you trying to accomplish with the poll?

I thought you wanted to figure out whether people want an overall score or not.

Yes, those in favor of having 5 scores for each category vote "categories". Those in favor of a one score system instead of five scores for five categories vote No categories.

EDIT: perhaps Mashek could change the title to include the words "voting thread" or such, so that nobody can complain they didn't get a chance to voice an opinion.

And Andrew don't worry about the vote counts, I will do it if needed.

[This message has been edited by matty12345 (edited 07-09-2008 @ 04:15 PM).]

how about you title the poll

category average vs. overall score

you can still assign an overall score and give scores to each category

Fine, if it makes you happy

Choose one of the following:

Vote: Averaged Categories

or

Vote: Overall Score

Don't worry the original 2 votes will be translated over.

If you're still worried about it not being seen, why is it still in this thread?

Make another thread for it. I'll link it in the topic post.

Yes, it couldn't hurt to make another thread. This is the discussion thread for new guidelines, a seperate voting thread would be justified.

Quoted from unnamed player:

which one is it? I'm inclined to agree with White Dragon on this point because that's what matches the guidelines. Fixation with coming across as a completely objective person is not needed. Humans aren't robots.

Taking Ulio as an example (we all know it), then I, a n00b reviewer, could easily award playability anywhere between a 1 and 5 because it is subjective and based solely on my enjoyment. Likewise with story, I can choose to rate the category a 4 because it was a story but not that great of a story. Balance is probably next, where a 4 is borderline but can pass. Map design and creativity and can still receive 4 or 5 ratings. If I rate playability a 2 because I did not like it, then the rating falls anywhere in between a 4.0 to a 4.4. A reviewer can mark Ulio down from being a perfect scenario to something quite average or a little above average, all in the context of the guidleines. Of course, I'm sure some reviewers could push to be harsher. Anyway, the situation I'm opting for is similar to your review for the Prophecy, unnamed player. We all rated the scenario highly (between a 4.8 and 5.0) where you rated it something like a 4.0.

OF COURSE, a reviewer rating any scenario, with any score, must not be discriminatory but honestly determine his opinion and the design at hand's merit. The example above (which is bad but'll do) doesn't necessarily reflect a bad review, but a harsh score, and can pass. So long as the reviewer can justify his ratings in a truthful manner and in the frame of the guidelines, then it'll pass. That's where I'm coming from.

I suppose where Luke is coming from is that someone couldn't award creativity or map design a 3 (such as in a great scenario as Ulio) for once of the greats as it just isn't justifiable. Creativity is exemplary in Ulio and map design is far better than a random map. A reviewer's opinion, experience and subjective choice of rating comes into hand when judging playability, balance and story/instructions - the latter two to a certain extent.

But I suppose they do have barriers to prevent marking playability down, (e.g. if you don't enjoy a particular genre you shouldn't play it - which I'm not sure I agree with).

[This message has been edited by White Dragon (edited 07-09-2008 @ 09:22 PM).]

Vote: Overall score.

Vote: Overall score.

Meh.

Vote: Averaged Categories

Creativity is exemplary in Ulio and map design is far better than a random map.

Yes, it is, but subjectively a reviewer could say that the design works against the scenario and it deserves a 3 or less (because a random map would have served it better).

I don't think that's true (Ulio is one of the few scenarios I'd rate a 5.0) but I think the guidelines could support such a position.

And if we are voting in this thread then:

Vote: Overall Score

Vote: Averaged Categories

I added a new topic and counted the votes already cast here.

Just to point out the people voting in this topic are a small fraction of the total number of download section users & reviewers. I don't know how representative this will be.

To clear a few things up - I agree the system has flaws and needs an overhaul. I agree reviewers need more flexibility but I don't think this is the way to do it; at all. It relies heavily on 100% of the reviewers being mature and responsible which just doesn't happen.

Designers need to be protected.

One thing I want people to consider is that a review is just a measure of a scenarios quality. It's not the ultimate end-goal for a scenario and there isn't some absolute fixed scale that every scenario lands on. The measure will fit some scenarios better than others, but it reduces the risk of obviously incorrect viewpoints passing muster.

When you sort by rating in the Blacksmith; you're sorting by how well the uploads fit HG's measure. The goal should be to make this measure encompass as many scenario types as possible. A free-for-all system as proposed would completely eradicate this problem, but from what I've seen would bring in so many other issues.

A question you should all ask yourselves is "What would I do / How would I feel if my excellent, extraordinary scenario got a 3.0 because someone didn't grasp the gameplay properly? (for instance)". Or because some 10 year old doesn't like getting attacked in the first 30 minutes. Even worse if that's the only review your scenario gets. That's what you're leaving yourself open to in a no category system.

I think people are debating the theory too much and not what happens in practice. Classic politician error.

[This message has been edited by Mascherano (edited 07-10-2008 @ 06:11 PM).]

I expect I'm being silly, but presumably the no-category system envisaged includes some guidance for the reviewer/moderator?

Quoted from Luke:

It relies heavily on 100% of the reviewers being mature and responsible which just doesn't happen.

Luke, I agree, but what about making the standards more stringent so that only mature and responsible reviews are accepted? I see that HG may wish to encourage user participation but even under the current system I see a fair few reviews which would wrankle me if I were the designer in question.

The stuff about Ulio getting a 3 seems like a complete red herring because it's just not going to happen.

I expect I'm being silly, but presumably the no-category system envisaged includes some guidance for the reviewer/moderator?

I don't know. Several of the arguments seemed to suggest no guidance at all but have changed to having some. Whether this would be in the form of 'rules' or 'guidance' I don't know.

Luke, I agree, but what about making the standards more stringent so that only mature and responsible reviews are accepted?

Whilst that would work, the issue would be how to enforce that with no categories.

For one thing it's extremely hard to define a mature/responsible review. I can't see "Sorry not mature enough" going down well with a 13 year old kid.

The other thing to factor in is that the guidelines would be much more difficult to enforce under a category-less system. Without categories there's no way of telling how much of what a reviewer has written he/she has factored into the score, or what the reviewer has deducted for that he/she hasn't written about. Whilst this obviously occurs under the current system there's much less scope for it to happen.

I just don't see how you can apply standards effectively to a category-less system. How do you apply rules when you could have little idea of how the final score came about?

Okay, I was looking in the vote thread, and can someone explain (or link / copy-paste to the reply that explains) the difference between the two? "Averaged Categories" and "Overall Score" sound like the same thing. Which is currently used? I mean I know how the system currently works, I'm just trying to attach the terminology to it's correct implementation...

=Matt=

Averaged categories give a rating based off of an average from all the categories. Overall score allows an 'overall score' to be given, despite the ratings of the individual categories.

Averaged Categories are currently being used.

I hope that helps clarify.

A question you should all ask yourselves is "What would I do / How would I feel if my excellent, extraordinary scenario got a 3.0 because someone didn't grasp the gameplay properly? (for instance)". Or because some 10 year old doesn't like getting attacked in the first 30 minutes. Even worse if that's the only review your scenario gets. That's what you're leaving yourself open to in a no category system.

The guidelines don't protect against these type of complaints. Moderators do. Whatever system gets adopted will have moderation.

I don't think "free-for-all" is what anyone is aiming for. An overall score approach would still have categories, but they may or may not be discussed in segmented blocks.

I think people are debating the theory too much and not what happens in practice. Classic politician error.

You present the situation as "free-for-all" versus "guidelines" and try to force people between two things that are not exclusive. Typical of politicians blowing things out of proportion and misrepresenting the other side along with invalidly appealing to people's emotions to get a point across and resorting to unsound practices to maintain police powers.

Jokes aside, I am aiming for a system that is both pragmatic and theoretically sound. There are a number of measures that have been proposed by various people that would accomplish this, but they have not been addressed or acknowledged. Perhaps it was just politician oversight...

Quoted from turty:

Averaged categories give a rating based off of an average from all the categories. Overall score allows an 'overall score' to be given, despite the ratings of the individual categories.

Averaged Categories are currently being used.

I hope that helps clarify.

Thanks, that makes perfect sense. But so then it appears to me that the vote is really NOT one of the major points of contention, it's moreso the guidelines and moderation of reviews that people want to change.

Sounds excessively political to me. Good grief...

Matt "Crasher"

I don't know. Several of the arguments seemed to suggest no guidance at all but have changed to having some. Whether this would be in the form of 'rules' or 'guidance' I don't know.

Actually at no point was it stated that we'd just scrap the guidelines and say FREE FOR ALL YAY. That's just your projecting, Luke.

As UnnamedPlayer says, under the current system a scenario could be marked down for exactly the things you describe. It's up to the review moderators to exercise judgement in both cases.

Oh, and it should be noted that an Overall Score system hasn't noticeably impacted on the reviewing quality in the Mod Pack category. They're pretty comparable to the rest of the scenarios.

Also, Turty got this wrong:

Averaged categories give a rating based off of an average from all the categories. Overall score allows an 'overall score' to be given, despite the ratings of the individual categories.

It's not 'despite the ratings of the individual categories'. An overall score system would do away with the category scores altogether. Reviewers could still use them in the body of the review if they so wished, but they would not be enforced as they are right now. With so much dissatisfaction with the categories as it stands right now, quite a few reviewers have expressed intentions to stop reviewing or cut down on their reviewing frequency.

Regardless of what's eventually decided, this is aiming to help reviewers. It's not a conspiracy to make the Blacksmith an anarchy.

[This message has been edited by Andrew Dunn (edited 07-11-2008 @ 04:58 AM).]

Actually at no point was it stated that we'd just scrap the guidelines and say FREE FOR ALL YAY. That's just your projecting, Luke.

At one point UnnamedPlayer IIRC suggested no review moderation at all.

Guidelines with no categories? Could you explain how that would work?

As UnnamedPlayer says, under the current system a scenario could be marked down for exactly the things you describe. It's up to the review moderators to exercise judgement in both cases.

This is my point - it's so much more difficult to do when you only have your personal opinion to back this up. You'd be removing scenarios purely purely because "I don't think this is right". Not good and whatever theoretical parallels you want to draw with "you do that anyway under the current system" don't play out like that.

Oh, and it should be noted that an Overall Score system hasn't noticeably impacted on the reviewing quality in the Mod Pack category. They're pretty comparable to the rest of the scenarios.

You do have a point - however its not as black and white as you're making out. You can view the modpack reviewing as a one-category system as for most mods there's only really one thing you can talk about, the visual quality or the data editing. You can't apply a similar single category logic to reviewing scenarios.

With so much dissatisfaction with the categories as it stands right now, quite a few reviewers have expressed intentions to stop reviewing or cut down on their reviewing frequency.

I've heard this argument a lot and from what I can see for the most part it comes from people who have barely reviewed at all. Whilst this may be true I'd like to put forward the possibility that it's a load of hot air and a lot of those people would not review anyway due to the time it takes.

Regardless of what's eventually decided, this is aiming to help reviewers. It's not a conspiracy to make the Blacksmith an anarchy.

I never said or implied it was.

I think it's misguided and it doesn't take into account the actualities of what happens in a DL section.

Anyway I've had enough of chasing my tail here.

Nevermind, I'm clearly blind.

[This message has been edited by Kor (edited 07-11-2008 @ 07:46 AM).]

At one point UnnamedPlayer IIRC suggested no review moderation at all.

I suggested no moderation of opinions related to a game's quality. My complaint was that you took a possibility that I suggested and attached it to something else entirely. This has nothing to do with the scoring system. People who support 1 holistic score may support moderation of opinions as well. And even if you remove opinion moderation, it isn't a free for all.

It's not 'despite the ratings of the individual categories'. An overall score system would do away with the category scores altogether. Reviewers could still use them in the body of the review if they so wished, but they would not be enforced as they are right now. With so much dissatisfaction with the categories as it stands right now, quite a few reviewers have expressed intentions to stop reviewing or cut down on their reviewing frequency.

From that, Mechstra, I can pull a personal, considerable summary that from the very beginning, the most prominent problem that desperately needs to be taken care of have been the current five categories themselves. Reviewers have been filled with dismay at the way they're implemented.

If the categories are indeed the main issue, why would a change in scoring system be indispensable? In my arguable opinion, we can simplify the matter just by disposing the current guidelines and enforcing a new, acceptable one with a whole new set of categories afterwards. As Luke stated, I fail to see how could a category-less guideline come into application. Reviewers have to facilitate their reviews with a thorough description, and they definitely need to base the scores they give on particular aspects, which in this case, are represented categories. They can't just base them all merely on personal interests or self-made, distinguished categories which in time would lead to a bias and subjectivity in the whole reviewing system.

Please accept my sincerest apologies if I happen to be putting obsolete debates on the table, but I'm just curious.

[This message has been edited by Andanu Trisatya (edited 07-12-2008 @ 05:56 AM).]

The reviewing system is already subjective. Reviews are, by definition, subjective.

So, you're saying that it doesn't matter when reviewers mark a direct 5.0 to a campaign because they won't have any categories to deduct the scores in?

Most people are prone to be tempted to give an overly positive review to one subject, based on personal likings, or just for bandwagon's sake. They have no need to base their opinions on a set of categories, say, they can just award five stars to a FF campaign merely because it was made in their favorite style and they had a great experience upon playing it, completely disregarding other primary aspects which can't be left unnoticed such as map design, while in fact it should have been deducted in terms of score. That is the 'subjectivity' I was talking about.

Categories could be used as a guidance, as well as restrictions, to a give well-deserved penalty to the overall score. Again, I don't see how a category-less system could work out.

[This message has been edited by Andanu Trisatya (edited 07-12-2008 @ 06:52 AM).]

The reviewing system is already subjective. Reviews are, by definition, subjective.

You're not wrong, but for the millionth time this has no bearing in reality. The amount of subjectivity is much much less under categories. End of.

Having categories of a review and forcing discussion of all the elements aren't necessarily the same. Score deduction should happen for important reasons, not arbitrary standards that have little relevance to how fun a game actually is. Categories tend to influence the reviewer toward the latter because someone will be drafting generic guidelines to encompass the entire range of games.

Categories along with getting rid of averages would be accurate enough, but after a certain # of approved reviews, people should have the option of structuring their reviews however they want while still talking about considering all the major aspects of a game.

[This message has been edited by UnnamedPlayer (edited 07-12-2008 @ 01:33 PM).]

I think I finally plump for the overall score because the current system forces the reviewer to give equal weight to each of the five categories.

Tigers and lions and bears! Oh my!

So, we want to say goodbye to the yellow brick road that has been the review feature? It has changed, and evolved overtime. I was one who was always working to maintain some measure of continuity with the past. However, changes can be good too. There are a lot of good ideas here that would radically change the feature. The thing I would be concerned about the most though is whether or not the changes could be moderated effectively. Moderation of Reviews has perhaps been the greatest improvement to the feature overtime. Still, I think the one rating idea is exciting, and I know it works from other files that are reviewed that way.

The inclusive and responsive thing might be to change the current two tiers of 'official' and 'member' reviews. Have the member one be the radical one rating system while the official one be kept to the more traditional category reviews. But make the official reviews more inclusive to include anyone who reviews to that standard.

There are pros and cons to both proposed rating systems. If we want examples we need only to look at the reviews that are done these different ways at the Blacksmith. Which kind of reviews seem to have the most value? Anyway, I think I can say with a fair amount of certainty that unless some requirements for what is considered, and commented on in reviews (the guidelines) it would tie the moderators' hands.

I've always valued the categories, and I am for "tweaking" the guidelines. So, here's a run down of tweaks and changes I propose for the attention deficit crowd:

1. No mandatory ratings (that stifle creativity) for any category. -- Do away with mandatory scoring requirements, and allow reviewers to rate a submission spontaneously according to their experience 1-5 using the guidelines. Make only par excellence comparisons and/or examples for all categories, and make these be of the game itself.

2. Do away with the requirement of a story for reaching excellent (no matter what scoring method is used). -- Games are not stories. Change the category to "Theme and Instructions." Then instruct reviewers to consider any story or history given as part of the theme, and to rate it along with the other considerations based on the merit of them. 1-5

3. Change the Multiplayer balance instruction to include the entire game, and not just the starting state. -- I think Spiney must have been thinking RMS when he came up with that?

4. Preface instruction to take into account the designers intentions in all categories. -- For moderation purposes, designers would be instructed to make their intentions clear by using the many devices of design, or otherwise be obvious.

5. Better instruct the balance category to include some explanation of difficulty dynamics, and mastery within a closed game system. -- If a game system is impossible, or you have to get lucky to win, then it is poorly balanced. Players are already instructed to take their skill into account, but a good balanced game allows average players who have played AOK campaigns and random maps to improve, while still providing a challenge for the very skilled player on the higher levels. The three levels of difficulty should be meaningful and provide noticeable differences in the level of difficulty if considered for balance.

7. Better instruct the balance category to explain choice balance. -- If one choice always wins while all others always lose, then the choice is poorly balanced, and is not a choice at all. That is luck again, and Spiney does already have an explanation of this. Still, if one choice is simply so much better than the others that no one would logically choose the others that is also poor balance. -- So, I think we could explain balance better and avoid reviewers deducting with the comment only that it was "too hard."

8. Preface instruction to emphasize that a review is not a contest or personal favorites list.

9. Preface instruction to remind Individual reviewers to remain consistent (from one review to another).

10. Guide the reviewer in the playability category to include replay value, conflict, challenge, pace, and tension. Give reviewers something to go on about when they are rating the Fun vs Bugs category. Seriously, the subjective consideration of fun is good and should remain as the dominant consideration for the category.

**************************************

So the first thing I would do is instruct the reviewer in the preface, that a review is not a contest and the goal is not to use the feature to indicate the best submission of all time. To emphasize that we should not attempt to set apart one submission from another with ratings, and not try to alter the overall average or ratings of other reviews.

Also, I would instruct reviewers to take into account the designers intentions for the design in every category--that is any intention made clear or obvious (like walls for a ship). Next, would be an instruction not to make a wish list of any category. Suggestions for how a designer could have improved often are replaced with a wish list for things the designer never intended. This is akin to the instruction not to make vague statements, and to answer more questions than we pose. Then instruct reviewers to be consistent from one review to the next. Last but not least, to emphasize that these and most other general instructions in Spiney's guide be taken into account for all the categories. So, a little consolidation of Spiney's points in the guide. That would be helpful especially if a one rating system was adopted.

There's been a lot of discussion here about the random map example in the guidelines. The random map comparison, example, and value judgement, should be removed altogether. In my view, one should be able to score 5 with a random map. Also the notion that pressing a button to create a map, makes the map less valuable, or even average is silly. Many of the same folks who argue this, would also argue that only the results matter, and it doesn't matter how much time and effort a design took.

Moreover, the bar for excellence in map design (if you will) is placed too high, not too low. As one of the most subjective categories there should be guidelines that place value on all approaches a designer can avail themselves to, and let the reviewer make a judgement between what is good, very good, excellent, whatever. 1-5 Note: The one rating approach might be helpful in removing my concerns? Perhaps by not forcing a reviewer to give equal weight to map design as a choice...

I also think reviewers ought to be allowed to take into account any perception of time and effort. If we thought, "wow that must have taken ages to create," that could be the positive that evens out a negative. Or if we think -- "What a creative idea to make a ship out of walls, it takes a stretch of the imagination, not my taste, but interesting nonetheless...no deduction."

Excellent could be any map that has the quality of a ES random map or ES scenario. AoK is an excellent, award winning game. That's where I'd start.

Again, I would guide reviewers to take into account the designers intentions for the map design. Like with historical scenarios, some designers try to recreate the historic landscape and others will take licence with it. Both are valuable approaches, and we should take that into consideration. Again, emphasize that a reviewer must be consistent with their own reviews. So, if a reviewer finds a random map to be just good or average, or whatever, then they should be consistent with rating all designs they review that way or give some good reason for the change.

In short we should make it so reviewers can make up there own minds, while removing some of the snobbery. To do this we should use only the game maps, both random and scenario for comparison, and as the excellent standard to be deducted from by the reviewer. Moderators would check a reviewers consistency. I don't expect that many would agree with me here about Map Design though.

I have a lot more to write about the subject, but I'll hold it there for now.

Stephen: Good reason, that is one of the pros of the overall rating.

[This message has been edited by Angel Anastasia (edited 07-12-2008 @ 07:30 PM).]

Excellent could be any map that has the quality of a ES random map or ES scenario. AoK is an excellent, award winning game. That's where I'd start.

Sorry, but no. That won't work.

Wont work to what end? Just so we understand each other, the sentiments are the same from my view point.

In any event, my idea is valid. Judging from your comment, the value considerations you'd make for map design are too high, too complicated, or worse, arbitrary. I've lived with the current mandatory value judgement outlined in the guidelines. So, I'm sorry too, but no, more of the same doesn't seem like it would "work" either. What are we trying to indicate? How much unlike, or some notion of "better" than the game itself a design can be? Or -- just to what degree a map is good?

Edit: I see these efforts to so called "raise the standards" as veiled attempts to make a contest out of the review feature. Good contests usually have time limits and other parameter restrictions to make them fair and exciting. Perhaps you guys want to make more of a pageant out of reviews but I don't. However, I'm willing to concede to the intelligent points made by the majority. This is not to say that I didn't appreciate getting my fair share of abuse in a forum debate -- Luke. I did, and I wouldn't want to miss it.

[This message has been edited by Angel Anastasia (edited 07-12-2008 @ 10:18 PM).]