OOookay.

Just keep in mind what the review guidelines are about.

Actually, it's okay.

[This message has been edited by Quaazi (edited 09-06-2008 @ 01:59 PM).]

Brilliant, Kor. You've done what most people thought was impossible. Well done also the staff members who actually implemented this.

For now, I'm happy with the current guidelines (but then I drafted a decent portion of them so no surprise there), but it will be interesting to see any new ideas that get raised here.

I read these guidelines again and I immediately thought of something that is worth to mention, regarding the playability:

When reviewers ponder the score, they naturally recall what was good and what was bad in the scenario, however, often forgetting what playability is all about: the entertainment.

I thought of a situation where a reviewer has played an extremely amusing scenario but remembers some flaws or minor lag in it; this should not be a reason to lower playability score if the player's entertainment attained remains high despite the lag or bugs.

There could be misunderstandings making people think that possible flaws directly affect the playability when, actually, possible flaws might affect the entertainment itself which then again affects the playability.

[This message has been edited by Powery (edited 09-06-2008 @ 08:48 AM).]

Yay, thanks Aro for actually implementing this ( and all those who contributed to the guidelines. )

Powery:

Yours is a good point, look at these two examples where I awarded the scenarios high points despite having bugs.

Seige of Osaka castle

The pirate outcast

[This message has been edited by Jatayu (edited 09-06-2008 @ 01:33 PM).]

Newswriters, gather around!

The new guidelines are a great improvement to the old ones, and worthy to celebrate! *orders cake*

Now, Aro realised that discussions went a little further than what is written in these new guidelines, and he suggested that we have an all-out discussion on the guidelines, which he asked me to kick off. This discussion is no holds barred (well, apart from those bits and pieces the Rules tell us about, so keep it civil!) and so you can even discuss adopting a different number of categories etc.

OK, I'll return the kick.

One important aspect that was missed in our first discussion was Moderation of reviews. That is I think all should understand the goals of the Review Moderators, and that they be made more transparent. In other words the new Review Guidelines, and all the changes to the guide will have little meaning if the Moderation is not consistent for all designs, and designers, reviews and reviewers. This has been one of the main concerns for Reviews in the community. What I suggest is that the Moderators create a mission statement that would also include a way to respectfully challenge their decisions on the basis of precedents.

I also think we lost an important aspect of the original guide when we removed Spiney's intention that we not review game types that we don't enjoy. Here is my suggestion for including it:

Quoted from New Review Guidelines:

Playability is probably the most subjective element of the scoring. It is simply a gauge of how much fun you had playing this particular scenario, in other words, an overall score of how enjoyable you found it.

However, there are many kinds of games, and what is fun for some might be tedium for others. To be fair we should not review scenarios with a style of play that we know we don't enjoy. For example: If we detest Blood scenarios then we shouldn't review them.

-- Also for the Balance category, we should make clear that reviewers only need to play a scenario once on one difficulty level to review. That is we could explain that using something similar to what Stephen Richards or myself proposed for the category guide. I think we should try to improve the Balance category by revisiting it.

-- Last but not least, I think we should tweak and improve the "General Guidelines" as the preface to the category guides. This is an important tool for the Review Moderators, and is essential in keeping reviews meaningful. Here is a quick draft of what I think the General Guidelines should be:

Quoted from General Guidelines Draft:

General Guidelines

Reviews serve a double purpose. They are written for both the scenario designer, and for the site visitor who is considering downloading the file. As such, reviews need to help the designer by noting aspects that are done well, and pointing out areas that might be improved. Reviews also need to provide enough information about the scenario so that visitors will know if the scenario matches their interests. Obviously, don't spoil the scenarios by revealing plots or secrets that should be discovered while playing, but let the visitors know what to expect.

Reviews must also take the design's original intention into account, and give the design the benefit of the doubt. If the design's intention is to provide a specific game type or experience, like a scenario that is mainly puzzles, or a map design that is obviously intended to be abstract (like a Blood map), then we must consider those intended aspects. It is meaningless to suggest that one type of scenario be improved by things that would only be found in another type altogether. In short, a review should help the designers improve the designs they have submitted, and not the design of the reviewer's dreams.

Reviewers may point to examples from other designs when suggesting improvements. However, a Review should never make a comparison with another design to justify a deduction or score. Keep in mind that Reviews are not a contest or a poll of our personal favorites. Also, while it can be helpful to note preferences for certain aspects of scenario design, no deductions should ever be made that could be considered merely a matter of personal preference.

Reviewers should also avoid vague statements, and not make a mere wish list with the suggestions for improvements. Obviously if a design is extremely undeveloped then more general ideas for improving would need to be suggested. However, suggestions for improving a design do not always warrant a deduction. If some minor improvement is being suggested for an area that is otherwise well developed, then a deduction would not be warranted. In other words, the more developed a scenario is, then the more specific in pointing out major issues that negatively impacted the design is in order. An important note here, is to not assume the role of director when reviewing.

Reviews are to be taken seriously, and are subject to removal by the Review Moderators. Reviewers must follow the Review Guidelines.

[This message has been edited by AnastasiaKafka (edited 09-06-2008 @ 02:22 PM).]

I updated the link in the main post Kor (I got rid of the "_new"). The link on the main Blacksmith page now redirects to the new guidelines. I also posted a news entry on this for all interested.

Keep in mind to those who wish to become official reviewers: "Any reviewer who has written 20 reviews (counting only those for scenarios, campaigns and unfinished projects) can be promoted to this position if they request this."

Welcome to Jatayu and Basse to the Official Review team by the way.

[This message has been edited by Aro (edited 09-06-2008 @ 03:08 PM).]

Thank you .

This is good news.

I'd love to be an official reviewer, but I'm still waiting for my first review to be approved...two weeks later! There really ought to be something about this in the revised guidelines...

As far as my two cents on the revisions...In several categories 'default' ratings for adding certain things have dropped down (story is an automatic 3, not 4; random maps are 2, not 3, etc.). This seems a little unfair to those less accomplished designers (though many of them could probably care less), especially for map design: what now seperates a horrible map from a bad map that's less than a random map? I don't like the idea of having "*.5" scores, because this becomes very limiting I think in terms of it becomes much harder to score 5.0 since reviewers can knock off a half-point for the smallest things. No matter what anyone says, a lot of people would love to get that 5.0 (myself included of course ). I would like to see a six-point based system, therefore. This would, I think, allow for the spirit of the changes mentioned above, while changing the letter to allow more space for the "bottom half" of scenario design. Obviously, the logistical requirements are staggering, but if anyone were to come forwards with such a proposal, I would be happy to support it.

[This message has been edited by YoshiX100000 (edited 09-06-2008 @ 08:42 PM).]

Heya Yoshi,
I have similar sentiments about the "automatic" ratings. I wanted them removed altogether. Like you I think there should have been more room made at the top, but not for the sake of fairness with regard to less accomplished designers, but for the sake of not limiting creativity.

Hey Aro,

I tried giving a dummy review today. Apparently the software still wants me to give points for the 5 categories.

I tried giving a dummy review today. Apparently the software still wants me to give points for the 5 categories.

Dang, I dropped the ball on this one. It's hard-coded into the Blacksmith that all reviewers, including Official Reviewers, must use the pre-set scoring categories on Campaigns. I was under the impression that there was a work-around through the Blacksmith's config file, but apparently I'm wrong. =/

I'll e-mail Pecunia about it so I can see if we can get it changed; in the meantime, any work-arounds anybody can think of (i.e. achieving a "category-less" review while still adhering to the system)?

[This message has been edited by Aro (edited 09-06-2008 @ 10:57 PM).]

The only thing I could think of is ignoring the "insert Playability rating here" prompts, writing one review with a final score in mind, then playing with the 5 categories to get them to average your score. If you were looking for a more technical solution, then I can't really help much.

The only thing I could think of is ignoring the "insert Playability rating here" prompts, writing one review with a final score in mind, then playing with the 5 categories to get them to average your score. If you were looking for a more technical solution, then I can't really help much.

No technical solution necessary; few people know how the Blacksmith is coded anyway.

And I was thinking of that type of work-around too, but the problem would be scores that didn't mean anything, which might confuse the designer. And if you adjust the scores to mean something, well... you'd be right back to where you started.

I sent out a mail to Pecunia. Hopefully we can get some clarifications; for now, I'll comment out the part of the new Guidelines that state official reviewers can review without needing to use the score system, since at the moment, it's a technical impossibility. (Sorry again for that. >.< )

Would you like me to sticky this, Kor?

[This message has been edited by Aro (edited 09-07-2008 @ 03:03 AM).]

I sent out a mail to Pecunia. Hopefully we can get some clarifications; for now, I'll comment out the part of the new Guidelines that state official reviewers can review without needing to use the score system, since at the moment, it's a technical impossibility. (Sorry again for that. >.< )

That's a shame. Perhaps just relax the guidelines on official reviewers in general for the moment, then?

Would you like me to sticky this, Kor?

That would be great.

Heya Yoshi,
I have similar sentiments about the "automatic" ratings. I wanted them removed altogether. Like you I think there should have been more room made at the top, but not for the sake of fairness with regard to less accomplished designers, but for the sake of not limiting creativity.

Sp that's another vote for a rating system with a 6 as a maximum attainable score?

I sent out a mail to Pecunia. Hopefully we can get some clarifications; for now, I'll comment out the part of the new Guidelines that state official reviewers can review without needing to use the score system, since at the moment, it's a technical impossibility. (Sorry again for that. >.< )

I was afraid of that, I don't think any other heaven even has a dual review policy. Well right now the 'official reviewers' part of the guideline sucks, it doesn't explain who they are or what they do.

Sp that's another vote for a rating system with a 6 as a maximum attainable score?

No, my comment was not a vote for a rating system with a 6 as a maximum attainable score.

Sorry, my comment about more room at the top was unintentionally misleading. Anyway, I am not a advocate of overall score, nor was I ever interested in dicking around with the point system. I always liked the 5 categories and the scoring method we have, only I liked them better when we didn't have to structure the review in any particular way.

[This message has been edited by AnastasiaKafka (edited 09-07-2008 @ 09:13 AM).]

Aren't here going to be TOO much so called sticky topics?
We got aload here now, normal forum topics are unnessesarily pushed downwards

It will stay pinned until we've resolved this situation.

Quoted from New Review Guidelines:

Balance:-

For scenarios with no interactivity, such as cutscenes, this category should be considered purely as another indication of Playability. This achieves a more accurate overall score.

I'm confused with this statement regarding rating cut-scenes in balance. Are we cutting the balance category from the review altogether, or are we still rating balance? If so, by what merit? Do we base balance on the aspect that the cut-scene is consistent with flow, or are we basing our judgment off the notion that the cut-scene followed the author's intent as rated previously? It would be pointless to retrace what was already mentioned in playability.

The statement as written means balance = playability score, so that playability for cutscenes has double weight.

A better solution would be to get rid of the bogus category and double the playability score if that is what you want to achieve. Calling it balance just confuses people. Giving a score to something that can actually relate to balance is just nonsense for cutscenes.

The statement as written means balance = playability score, so that playability for cutscenes has double weight.

That seems a little pointless. How about a new category other than balance altogether? As matty mentioned earlier in the previous thread? (e.g. rating on consistency of flow)

One huge way to measure balance in cutscenes that everyone misses is the text! Is the text long and boring history/life stories, or is it too scarce to make any sense? Is the amounts of text vs action balanced? Etc.

That seems a little pointless.

it is imo

That seems a little pointless. How about a new category other than balance altogether? As matty mentioned earlier in the previous thread? (e.g. rating on consistency of flow)

Outside of the two of us, my idea wasn't terribly popular.

Well, I really don't see why not. It was a good idea.

And if not flow, how about something else more central to cut-scenes? Like...

what you should be doing is figuring out what you want to say is universally important for custscenes.

for some reaosn, you seem to be trying to trying to force something somewhat related to the current categories, and making 5 of those as equal weight.

there's nothing that says a game needs 5 components. cutscenes aren't even games. make it 4 if that's all that matters. nobody will rage. playability shouldn't even be called playability since there's no playing.

Perhaps a category such as ENTERTAINMENT would better justify a cut-scene than playability.

So we have four categories: Entertainment, Creativity, Map Design, Story. Personally, I think this might be a good area to introduce a 6-point system, except for the fact that it wouldn't fit in with the rest of the reviews . In a cutscene, since the player does not play all of these aspects must be enriched. a 5.0 would be a cutscene of the same quality of a 4.7-5.0ish scenario/campaign; a 6.0 would then represent this enriched quality; am I right?

cutscenes should have their own category separate from campaigns and scenarios

It would be pointless to retrace what was already mentioned in playability.

The idea for repeating the Playability score is to be able to give a score for the non applicable category (Balance) that will best reflect the experience without changing the software.

The other idea (in use) is to give an automatic 5. There is merit to both, but using the Playability score penalizes (potentially) the designs more than using an automatic 5 rewards them.

So the goal is to come up with a way to fill in the Balance score so that we can Review cutscenes.

I favor the automatic 5, because when we looked at the math in the past it made the most sense -- in that it worked best to keep the intended hypothetical scores we tested.

[This message has been edited by AnastasiaKafka (edited 09-13-2008 @ 07:15 AM).]

Why not just give balance score depending on the average of all other categories? With it we can minimize the margin of error without too much work. (Anybody here who doesn't know how to count an average from something?)

Automatic 5 isn't fair solution. It affects too much when it comes to low-quality cutscenes.

I know I'm late to the party, but I just want to put my two cents in.

I like the "points" system as it's structured. I've found the scores to be very accurate, and they help me download the best campaigns.

If a score is lacking in a particular area, that is also a great help to me. For example, if a campaign is lacking in story, I know I probably wouldn't enjoy it too much. If it's lacking in balance, then I may be more inclined to give it a shot. Just examples.

Not sure if you guys are aware of this yet, as I'm stumbling in here, but a possible cut-scene category (files submitted as a cut-scene; AFAIK, the scoring programming is category dependent) could have it's own set of guidelines and scores created. This would be similar to how RMS files are scored with their own unique guidelines, and it'd be no problem at all as long as you guys composed something that everyone would be happy with.

Since it relates to what kind of category a file is submitted as, of course, it'd have to be something Tanneur approves of.


Edit -- revisions, to make sure you understand what the hell I'm talking about. - Aro

[This message has been edited by Aro (edited 09-16-2008 @ 09:20 PM).]