Hey,

if something is good enough, why would you change it?

I have no idea what we should change about it, imo these are good guidelines and I don't think a reviewer needs more (I'm a reviewer myself...)

Greetings,
Guthan,
Reviewer

The problem is that they're not good enough. They don't allow for enough 'reviewer's discretion' and are too restrictive in what marks should be awarded.

There's been a lot of controversy about that in the review thread.

As you can guess, I'm in favour of change. Someone should contact Tanneur, if they have a reliable method of getting to him.

You forgot to post what the problems are and the change needed. To quote Andrew and Enraged Orange:

Quoted from Enraged Orange:

Quoted from Tanneur99:

We do not moderate opinions as long as they are in accordance with the guidelines and the reviewer gives a reason for his rating, especially for utilities and mod packs because there are no rules apart from giving a reason for the rating. I must constantly allow other opinions as unpleasant they might be for me. My objectives as a moderator are fairness towards the designer next to informative for the player and according to the tutorial, but the standards of fairness as a moderator can only be in form and content, the review moderator needs objective reasons for his moderation. Technically unfair would be if a reviewer devalues an AoK scenario because the designer did not rename units, which is not possible with original AoK, devalues an AI because it cannot hunt boars or if he rates balance down for being too easy complaining that the enemy did not attack his villagers (he obviously played standard).

However, if a reviewer proposes a mix of desert, dirt 1, dirt 2 and dirt 3 terrain for a desert map or writes that a winter map contains too little eye candy and proposes forage bushes and flowers then my hands are tied as a moderator. The Review is an opinion, maybe a distasteful opinion, however, as a moderator I have to let it pass. As a moderator one is in a constant conflict between approve and remove and if one's own opinion plays a strong role, you cross the line to censorship. After this excursion of why I first approved the review of Renaissance, now to the reasons of retracting the rating.

This is, in my opinion, exactly how you should handle things. However, my experience with my last review and comments you've made in this thread make me question whether this really reflects the actual situation. I posted this in the Blacksmith thread a couple months ago, where it was apparently overlooked - I'd appreciate it if you took a look at it, as it's related to the discussion here:

Quoted from Enraged Orange:

Just noticed this thread (I have this bad habit of scanning past stickied threads for some reason). I am glad to see there are other people who are unhappy with the current review system.

I agree that simply rating five categories and averaging them to reach a score is a horribly insufficient method. For an intricate RPG, story can be enough to make or break a scenario, whereas in some build & destroy scenarios, the scenario is at best an afterthought. Likewise, "balance" is a difficult category to be subjective about in general, and really not that big a deal most of the time - scenarios shouldn't be too easy or hard, but it usually takes a very large degree of error in one direction or the other to actually significantly detract from the scenario. Some are easier or harder than others, big deal.

Since there's so much oversight of reviews anyway, why are these strict categories even necessary? Why not simply list them as suggested areas to focus on, and let the reviewer base his score on whatever criteria he feels are most significant, rather than some strictly-mandated (and, at least in the 6+ years I've been here, never agreed upon by community consensus) categories?

Even within the current scheme, there's very little leeway given to reviewers to make their own objective assessments. Reviews which are well written can still be rejected because the "reviewers of the reviews" simply disagree. As an official reviewer and the former maintainer of the forums review thread, I think it's fair for me to consider myself as fairly well aware of these almighty guidelines. Yet my last review (and first in years), of Ingo's When the Wheel Breaks, was rejected because my "text is inconsistent with [my] rating. [ I ] critique the scenario in [my] review and still rate it with an overall 5.0." WHAT? So only a flawless scenario can receive a 5.0 now? I assure you, no such thing exists in the blacksmith. Could you imagine the Chicago Sun-Times withholding one of Roger Ebert's reviews from publication because he "critiques the film in his review and still rates it with 5 stars?" Any decent review should include criticism! What use to a designer is a fawning couple of paragraphs extolling his work, without any suggestions or critiques? 5.0 does not and cannot equal perfect - it should indicate that, in the opinion of the reviewer, the scenario (or whatever) was extremely well crafted and has no flaws significant enough to impact one's enjoyment of the scenario. These "critiques" in my review were things like "When the Wheel Breaks is a relatively minor work," and "...simply countryside and small villages...there is nothing revolutionary about the map...it serves its purpose fine." As far as I am concerned, a scenario not being of an "epic scale" or not employing every silly map trick in the book makes absolutely no difference in my enjoyment of it - certainly not enough to warrant decreasing my score in that section by 20%!

Looking over reviews to make sure they're substantive and accurate is fine. However, a review (of anything) is first and foremost the reviewer's opinion of a work. This should come through in the review. I feel that the 5 category system is entirely artificial and does not accurately reflect the quality of every scenario. But even if we stick with it, it is an enormous overstepping of the authority of the blacksmith reviewer to gauge for himself what score corresponds to a review - that is very much the responsibility of the reviewer.

Sorry to go on so long about this, but I feel pretty strongly about this, and it was the "rejection letter" of my last review, which I found very condescending and insulting and went so far as to imply that I was being dishonest in my review, that caused my most recent lengthy absence from AoKH. I hope the powers that be at least give this some consideration.

Quoted from Tanneur99:

In addition, you wrote for Map Design “…still better than a random map...slightly better than a random map.” Slightly better than a random map rates 4.0 according to the review tutorial. I changed the map design rating to 4.0 to make it consistent with your text.

This is a perfect example of the flaws in these rigid guidelines. Suppose the map in question had been a randomly generated map with a nice little grove of trees stuck in one corner? That's better than a random map, yes. But does that really seem like something that should warrant increasing the map score by 33%? What compelling reason is there not to leave that kind of decision to the discretion of the reviewer? I was a fairly prolific review writer years ago when these guidelines were, for the most part, guidelines - not laws. I certainly would not write any reviews under these conditions, where someone feels he can take the liberty of "changing ratings" on someone else's review, or, in the case of my aforementioned review of When the Wheel Breaks, informing me that my scores (that I awarded) don't correspond to the review (that I wrote) and that I must amend my error. That is patronizing and unnecessarily bureaucratic.

Quoted from Tanneur99:

The veteran players supported the guidelines, unfortunately I am the only one left.

I question the relevance of this; opinions change, and the guidelines do not appear to be supported any more. That aside, I assume I fall under the "veteran players" category. I do not support the rating guidelines, as you may have gathered. And yes, they were generally supported in the past - I had no problem with them - but they were far less draconian then than they are now. Random maps got a 3, lack of story capped Story/Instructions at 3 (or else 2 - it's been a while), and cutscene scenarios received the Balance score which least affected the overall rating. Beyond such basics, the reviewer was allowed to rate as he saw fit (within reason).

Quoted from Andrew Dunn:

I also don't support the review guidelines, but then I'm a pretty schismatic heathen in these parts when it comes to ratings because I've come to the conclusion that we shouldn't have categories to rate at all. Or 'point scales'. A simple 1-5 scale without any decimals would be far better, in my view.

This is going on the assumption that we can't do away with scores altogether, which would be my ideal system. The onus would then be on the text of the review rather than the score it receives.

And it would solve once and for all any debates about the intricacies of what deserves what score, or how much balance and playability overlap, and so on. Right now categories are often artificially inflated to get an overall score that fits the scenario better - think of how many scenarios with a '5' rating for creativity aren't very creative at all, but are excellent examples of previously existing ideas? Just because a film is a 'genre piece' and therefore not entirely original doesn't make it a bad film, and the same is true of scenarios.

But then this is a pipe dream, seeing as even small changes to the guidelines in tune with the desires of the community seem to be anathema to The Man.

EDIT: I will back up EO (whoa, those initials are bringing back memories of another SDer) that the guidelines weren't as strictly enforced in the past, and I agree that this was for the better.

[This message has been edited by Impeached (edited 06-29-2008 @ 01:05 PM).]

why not just let us to be able to choose .5's, it would make it easier.

E.g 3.5 for playability.

That's exactly the opposite of what we should do. We should be moving away from step-by-step decimals and towards simpler ratings.

[This message has been edited by Andrew Dunn (edited 06-29-2008 @ 02:45 PM).]

They are called "guidelines" after alll; but they are enforced to a level that doesn't allow the reviewer to give a balanced and thought out review* that is beneficial to the creator and player.

*(You could argue that under the current system it is balanced. But it is in no means fair to the reviewer for the reasons Orange has spoken about, namely the random map with extra grove of trees possibility. Plus there is also the area regarding the Story/Instructions area that doesn't allow the merit and content of the story to be marked, for as long as there is even the smallest shred of some story or background information to the scenario you will be given a high mark in that area, at the moment there is only the difference between "A story" and "A good story" when it comes to makring)

That's just my personal opinion.

The current implementation of them treats them as more a law than a guideline, and the standards to which they hold scenarios are eight years out of date, if not more (as they were first drawn up for Age of Empires 1 scenarios).

Can we seriously say that we still think of a random map as 'average' map design, in this day and age? No. Standards have advanced. But that doesn't mean that in certain situations a random map does not deserve a 3, or even a 4. It depends on the scenario. How it fits in.

It should be up to the reviewer to decide.

[This message has been edited by Andrew Dunn (edited 06-29-2008 @ 03:17 PM).]

I am in full agreement with you. The review guidelines MUST be changed. I'm glad action has finally been taken!

We are on the march again eh?

Ok, here is my two cents:

The review guidelines are there to serve the DESIGNER and the DOWNLOADER. The designer is ensured a fair review, and the downloader is given something to go by that (should) consistently tell him about what quality to expect in the download. So in that sense, the guidelines are their to serve the community.

Now, the community dislikes the guidelines. We think they are outdated and we want changes. Since they are there to serve us, why should we be not allowed to change them?

They are there to be fair to all designers, to ensure each review is fair with the others. Well how is it fair that WTWB gets a 4 for story (incomplete, and perhaps a fair deduction), and any goofy story is automatically a 4 NO matter what. In this sense and many others the guidelines are failing the designer.

And they do a disservice to the downloader as well. Why should I give virtually all cutscenes a free five for balance, when I can tell the downloader something actually meaningful if I create a new category for cutscenes? In this way and others the guidelines fail the downloader.

If something is failing to do its job, you change it.

I think the fairest solution is to have a general discussion to rewrite the guidelines to the community's satisfaction. For this to happen though, we need someone to actually join the discussion instead of pretending it doesn't exist.

I firmly believe that the sole reason these posts are ignored is it will create extra work for those in blacksmith moderation. That is a horrible reason to resist change.

Good posts; Matty, especially, makes the point very eloquently.

One extra point I'd like to add, which I haven't heard before: either make new official Blacksmith reviewers or remove the previous ones. At the moment there are a few (now basically entirely inactive) official reviewers, but their reviews are so old that if one looks at the official Best of of the Blacksmith there's only one file from 2008 and one from 2007 there. What is particularly ironic or even galling here is that the 2008 file is Medivia, a joke campaign. And even far-off 2006 is represented with only seven projects.

This means that the Best Files section no longer accurately represents the real best files of the Blacksmith. Certainly, the projects there are of very high quality, but there are more files out there that can join those ranks. This can be done quite simply by elevating some of the high quality reviewers we have to also have official reviewer position. While not everyone can reach the 90+ reviews BrandNewCar has (let alone the almost unbelievable 230+ of Tanneur!) not all people presently acknowledged as official reviewer have achieved that much, anyway - for example Cat has only six reviews, Gordon Farrell has five, and thurdl01 has only two!* Something tells me they received their elevated position more because of their position within the design community (Cat as a very active playtester, Gordon as a top designer, thurdl01 because of his staff position) than because of their quality reviews.

We have a new generation of reviewers and a new generation of designers, and promoting active reviewers to a more exalted position is one the best ways of recognising their achievements and stimulating reviewer and designer alike to carry on the work.

* This is after a cursory glance - there may be more reviewers with similarly low productivity.

This is in a separate thread because it keeps being ignored by the mods (and Tanneur in particular) in the thread it's most suited to, the Review sticky.

The Review Guidelines, as they are, need attention and change. The community is mostly agreed on this, particularly reviewers such as Julius (who has said he'll give up his reviewing if the current situation continues) and many of the oldies who are still around.

Now, let's not get carried away. You make it sound like this is some flag-waving protest march with Tanneur as the evil bad guy. I'm all for changes, but there's no need to demonise the mod staff and turn this into a Greenpeace rally.

As you can guess, I'm in favour of change. Someone should contact Tanneur, if they have a reliable method of getting to him.

I'm pretty sure he knows these complaints exist. But for some reason, he's not addressing them.

One extra point I'd like to add, which I haven't heard before: either make new official Blacksmith reviewers or remove the previous ones. At the moment there are a few (now basically entirely inactive) official reviewers, but their reviews are so old that if one looks at the official Best of of the Blacksmith there's only one file from 2008 and one from 2007 there. What is particularly ironic or even galling here is that the 2008 file is Medivia, a joke campaign. And even far-off 2006 is represented with only seven projects.

I agree. Having Official Reviewers is a nice idea, it means you can separate the 5.0s awarded by reputable reviewers from the 5.0s awarded by random wackjobs who, for all we know, might be the designer's brother. But it kinda defeats the purpose if there are no Official Reviews actually writing reviews.

Something tells me they received their elevated position more because of their position within the design community (Cat as a very active playtester, Gordon as a top designer, thurdl01 because of his staff position) than because of their quality reviews.

Or perhaps their willingness to co-operate with the guidelines. I remember that Berserker Jerker wrote a review that fulfilled the guidelines exactly, and Tanneur put him on the Official Reviewers list with no questions asked. Now, BJ was a great reviewer and I'm not saying he didn't deserve to be there, but how did Tanneur know that BJ wouldn't just vanish again without ever writing another review? Would a person looking for a secretary hire you after seeing you take one phone call?

Let's keep one thing in mind guys. Tanneur99 is one of the hardest working, if not the hardest working mod at AoK Heaven. He deserves respect, so let's not let this degenerate into flaming like some other times this topic has been brought up.

Now, let's not get carried away. You make it sound like this is some flag-waving protest march with Tanneur as the evil bad guy. I'm all for changes, but there's no need to demonise the mod staff and turn this into a Greenpeace rally.

Why not?

It is abundantly clear that Tanneur99 not only refuses to change review guidelines, but treats the whole issue as if it does not exist.

Other mods eg. Mashek ( others are inactive ) and 'official' reviewers like EO ( and a lot many other reviewers ) have supported the issue.

There's no point in denying such a markedly obvious thing, although 'some flag-waving protest march with Tanneur as the evil bad guy' is a bad way to put it, but nevertheless correct.

Making ridiculous excuses such as 'he's busy' and 'he's hardworking' will not help. This has gone on years. It's time to call a spade a spade.

Let's keep one thing in mind guys. Tanneur99 is one of the hardest working, if not the hardest working mod at AoK Heaven. He deserves respect,

This is exactly the kind of politeness and political correctness that sabotages these kind of discussions.

People just refuse to believe certain things when they are staring them in the face.

I agree. Having Official Reviewers is a nice idea, it means you can separate the 5.0s awarded by reputable reviewers from the 5.0s awarded by random wackjobs who, for all we know, might be the designer's brother. But it kinda defeats the purpose if there are no Official Reviews actually writing reviews.

In theory that is correct, however
1. There are no active 'official' reviewers
2. The reviews written by 'official' reviewers are old and out of date. A few months ago I had written about pathetic scenarios getting 4.6, 4.8 under 'official' reviewers. Simply put, the best of Aok category no longer represents the best of Aok.

Another thing to mention: the 'top rated' category is also faulty, as it is fairly obvious that 4.6s of today and 4.6s of 2000 are completely different things. What should be done is a list if all files with rating > 4, sorted by date.

I for one rarely review single player any more, only modpacks and utilities. It's easier and there are no 'guidelines'.

What happened to the official complaint against Tanneur99?

Edit: How often do we have this discussion btw? Once every 3 months?

[This message has been edited by Jatayu (edited 06-29-2008 @ 09:36 PM).]

This is exactly the kind of politeness and political correctness that sabotages these kind of discussions.

People just refuse to believe certain things when they are staring them in the face.

There is nothing wrong with believing something while presenting it politely. Insulting people while presenting your points usually goes nowhere.

What happened to the official complaint against Tanneur99?

Tanneur99, despite ignoring this issue, is near irreplacable and is still great at his job. I for one would rather have Tanneur99 than a change in the guidelines, if one had to be chosen. I can't think of anyone who could replace Tanneur's diligence and hours spent on approving submissions, comments, and (though less time would be spent on these if changes were made) reviews.

--

Anyway, I agree heartily that the review guidelines as they are are completely out of date and too strictly enforced. Tanneur's last post in the review guidelines thread said that all the veterans would agree with him, but they are gone. Now Andrew Dunn and Enraged Orange ('veterans') post, disagreeing.

[This message has been edited by Impeached (edited 06-29-2008 @ 10:12 PM).]

Quoted from matty12345:

I firmly believe that the sole reason these posts are ignored is it will create extra work for those in blacksmith moderation. That is a horrible reason to resist change.

Eh, I'm not so sure. After all, loosening the review guidelines would actually make less work for the blacksmith moderator(s). What Tanneur's reservations may actually be, however, I have no idea.

Quoted from Jatayu:

This is exactly the kind of politeness and political correctness that sabotages these kind of discussions.

I guarantee you that shifting the focus from the issue at hand to a witchhunt against Tanneur will sabotage this discussion more than treating everyone involved with respect. Tanneur is an extremely hard working and dedicated member and despite any complaints we may have with this system it's important not to lose sight of that. Sure, we shouldn't just consider him infallible and yield to his judgment without discussion, but it's the system, not Tanneur, that has a problem here.

The biggest problem is with the theory behind the guidelines itself. Nobody can breakdown how much weight each element of a scenario should have. It is not because some people think one aspect is more important than another, but because the importance of each scenario element depends on the individual game. As much as damage control as guideline defenders try to do, story really does not matter in a multiplayer scenario. The whole explanation about playability and balance in cutscenes just proves that things matter differently for different games.

Scenarios are just different types of video games, and AoKH's review system is terrible and archaic when you look at video game reviewing as a whole. You won't be able to find a single credible reviewer who can give a strict weighted formula to rate games. The only thing you might find is people who say enjoyment determines 100% of the score. This applies to all entertainment mediums.

The reason for this is that each game or book or movie is more than the average of its parts. Each is a unique art form judged based on how good it is, not how well it conforms to certain established guidelines. When you look at reviewing, things are rated based on their genre. You don't criticize a Super Mario game for not having enough strategy, and you don't criticize Age of Empires for not having enough platforming.

The same thing works with scenarios for AoK. The only person who can determine how important something is in a game is the player (who is also the reviewer). The point of review guidelines is to guide reviewers, not do their job for them. All the mediocre video game reviewers can figure out scores for categories, but only the good ones can determine which flaws to disregard and which parts to reward.

This is subjective and varies from game to game. The fundamental problem with AoKH's review guidelines is that they turn something subjective into something objective. In this case, the methodology by which we must maintain consistency in reviews eliminates the ability to accurately judge the quality of a game.

The people in charge want reviews to be consistent. What that means is that every game that does not break down into the 20-20-20-20-20 ratio is reviewed incorrectly.

Why are you so passionate about this subject? It's because you want your score to accurately reflect the quality of a scenario. Adding half points changes the 5 point scale to a 10 point scale which is mathematically more accurate, and therefore, more in line with your wishes.

Another proposed solution is to have separate guidelines for everything. This solution contains the same theoretical flaws as the original guidelines because it confines reviewers to a certain ratio. Even if the ones in charge decided to craft 10 different sets of guidelines for different types of games, they would be disputed because it would still be changing something subjective into something objective.

What makes the guidelines author's opinion better than a reviewer's opinion of how a game should be scored? By agreeing with the guidelines, you're essentially saying, "The guidelines author is worthy. I am not. His opinion is right. Mine is not. He knows how to rate a scenario without even knowing what type of game it is. This method is better than what I could come up with after actually playing the game."

The only remaining defense of the guidelines is that consistency should not be sacrificed in favor of accuracy. I disagree with this position, but some of you may not. The point is that the review guidelines fail in this respect also. The point of the reviews is to help the AoK community - the downloaders and the designers.

In their current form, the guidelines only help their author. All our reviews give him opinions on which games to download because we're using his rubric for grading them. When the rubric isn't even accepted by the community then what do the guidelines accomplish? They fail to provide accuracy and they are consistent to something people don't even agree with.

Yes, the guidelines "work", but so does my Pentium II computer. That does not mean I should still use it when there are better machines around.

the system, not Tanneur, that has a problem here.

Point is that the problems with the 'system' have been discussed and repeated ad infinitum. But there will not be any changes so long as Tanneur99 does not agree.

Theoretically the changes in review guidelines ( or anything ) follows this process

Community criticizes system -> Person in charge listens -> Changes the system -> Community tries new system -> Step 1

We have already passed step 1 long ago. Step 2 is the bottleneck.

All attempts at trying to get the person in charge to listen have failed. Therefore the only rational way to change system is to change person in charge.

I know, I sound like a communist.

........

For Tanneur--- I know he's reading this, here's a quote from matty.

Whenever you are alone in something, you are either a true revolutionary ot just plain wrong. Decide for youself where you stand.

[This message has been edited by Jatayu (edited 06-29-2008 @ 10:37 PM).]

OK Che, sure, let's just gather up our torches and pitchforks and run Tanneur out of here. Problem solved!

Oh wait, hmm, now we don't have anyone to moderate the blacksmith, we've lost our most prolific reviewer and one of our longest-tenured and most helpful forumers.

I'll pass on that approach. Tanneur's a great guy and just because he's being stubborn or whatever on this one issue isn't grounds to start clamoring for his replacement.

OK Che, sure, let's just gather up our torches and pitchforks and run Tanneur out of here. Problem solved!

Oh wait, hmm, now we don't have anyone to moderate the blacksmith, we've lost our most prolific reviewer and one of our longest-tenured and most helpful forumers.

Look, I don't have a solution to the problem.

But there is no point discussing this if it falls on deaf ears.

Jatayu, this thread is for discussion of the guidelines only, and Tanneur was mentioned because he is the mod this should be focused on starting discussion with. And because he has the most influence on these matters at this Heaven.

As I always said, We should be allowed to give .5 ratings like 1.5, 2.5, 3.5, 4.5 etc. I am not asking for a complete decimal system, but a .5 system. If we could atleast decide between a difference of 0.5, that would make our reviews much more accurate.

I think the review guidelines are ok as long as moderators dont impose it as a rule. Let it be just a basic guideline. Give us some subjectivity. Tanneur himself has slight misunderstandings about the guidelines, it is all ok if he agrees (or if he already has) to let guidelines be 'guidelines'.

[This message has been edited by John Mendl (edited 06-30-2008 @ 08:03 AM).]

I have some quibbles with the reviewing guidelines and I agree with Kor's comment about the official reviewers - the only thing here which has always irked me - but I think the onus is on those proposing change to set out their manifesto clearly and succinctly.

Are you proposing a complete overhaul? Or do you want to tweak certain aspects? If so, which? It will be difficult to find a system that pleases everyone which is why the current guidelines may not be as bad as you think.

I say tweak. Let's not run before we can walk.

Let's not get too militant either. People don't listen to a bunch of revolutionaries, especially when they're not compelled to by the threat of death. No need to sacrifice courtesy.

We could compile a list of suggestions and then work out what the majority thinks is the best route to go down. Here are a few:

1. Abolish the automatic 4.0 for a story, any story. Have it as 3.0 instead.
2. Change the automatic 3.0 for something equal to a random map to an automatic 2.0.
3. Appoint more official reviewers.

[This message has been edited by Julius999 (edited 06-30-2008 @ 10:10 AM).]

1. Abolish the automatic 4.0 for a story, any story. Have it as 3.0 instead.
2. Change the automatic 3.0 for something equal to a random map to an automatic 2.0.
3. Appoint more official reviewers.

I can go for all of these.

As I always said, We should be allowed to give .5 ratings like 1.5, 2.5, 3.5, 4.5 etc. I am not asking for a complete decimal system, but a .5 system. If we could atleast decide between a difference of 0.5, that would make our reviews much more accurate.

No no no no no.

I too could go for all those, though it's worth noting that any continuity with older reviews will be lost. However, AoKH is not a democracy and is not obliged to listen to anything anyone says!

1. Abolish the automatic 4.0 for a story, any story. Have it as 3.0 instead.
2. Change the automatic 3.0 for something equal to a random map to an automatic 2.0.
3. Appoint more official reviewers.

I agree, especially seeing as hardly of the official reviewers review any more.

1. Abolish the automatic 4.0 for a story, any story. Have it as 3.0 instead.
2. Change the automatic 3.0 for something equal to a random map to an automatic 2.0.
3. Appoint more official reviewers.

These are the very least, but everyone agrees on these.

I believe one of the categories could be considered ambiguous: "story." It is always one that has irked me slightly, especially in regards to multiplayer maps, personally I feel it is one that should not be considered for any multiplayer map except for those that deign to include one, even then it shouldn't be judged upon by the same standards of a SP map, simply because we will not see cutscenes in MP, it wouldn't make sense to have a 5 minute cutscene, and if you want to have a story that really grabs the players attention and allows actualy development of plot and cahracters, that is what you would need. It doesn't work on online, and I know this for sure!

Some of the best [and popular] Mp maps are Spider and Castle blood etc. Yet are rating system means that I would have to give a lower mark for "Story/Instructions" for that reason.

Of course instructions is another one, some designers nail it perfectly in SP, but in some MP maps, it should be a simple case of, kill the people who aren't with you. Of course they could include brief hints and an explanation of power up zones etc.

1. Abolish the automatic 4.0 for a story, any story. Have it as 3.0 instead.
2. Change the automatic 3.0 for something equal to a random map to an automatic 2.0.
3. Appoint more official reviewers.

For starters, this is good.

I'm currently drafting an amended Review Guidelines, incorporating these two relevant changes and correcting a few typos and changing a few other things, which I will highlight when I post the draft for discussion. I'll also post my 'ideal' copy, which is a pipe-dream but might give some indication of other areas we can look at for improvement.

Dunn i dunno why u r opposing something aimed at accuracy. .5 system is simple enough. There are cases when 3 is too low and 4 is too high. 3.5 is appropriate there

I'm against it because it's needlessly anal and muddies the waters even more. You'll end up with scores like '4.25', and what the hell does that benefit anybody? Quibbling over what exactly constitutes a .5 increase in quality in a single category is ridiculous, even more so than the already ridiculous incremented categories. Which I accept we won't be able to get consensus to change.

Anyway, as promised, here are the redrafted review guidelines. Anything in bold is added or changed, and strikeout indicates bits I think we should remove. The whole of 'Balance' is in italics because I'm really not happy with any of that category, and I think we need to look at it in-depth (particularly the assertion that a well-balanced scenario shouldn't be beaten first time through - absolute nonsense, in my view).

Some of the smaller changes are typo corrections and improvements in style (fewer ellipses all over the place), and the substitution of the first-person 'I DO THIS' perspective for a more neutral royal-we approach. See what you think. This is all up for discussion, so pick out bits that you feel should change too, including anything I've written.

Playability

Playability is probably the most subjective element of the scoring. It is simply a gauge of how much fun you had playing this particular scenario. One thing to look out for when reviewing is to only play scenarios that use a style you enjoy. For example, if you hate playing RPG scenarios, don't try to review one since you are bound to not enjoy the scenario. Try to keep within styles that you enjoy.

There really are no specific criteria on how a score is given in Playability but there are quite a few things that can effect playability in a negative manner. Trigger bugs, victory condition bugs and any other playability-destroying bugs obviously can ruin a scenario's playability. Lag is another playability issue that a scenario can be marked down for. If a player is ever confused about the next goal to accomplish, that's a playability problem. If a player can complete an objective in a way that the author obviously did not intend to be possible (i.e. there's a hole in a wall that allows the player to skip half the scenario), that's a playability problem. Anything that adversely affects your enjoyment of a scenario can affect the Playability score.

Balance

Balance is also somewhat subjective since each player is a different skill level and what might be perfectly balanced for one player, might be way too easy or way too hard for another. As a reviewer, you must take your own skill level into account when giving a balance score. A perfectly balanced scenario should provide a challenge for a veteran player. Most people who are downloading scenarios from the internet have at least played through the campaigns included with the game and have a good knowledge of the game.

Most perfectly balanced scenarios should not be able to be completed without the player losing a few times. If a player is able to complete the entire scenario the first time, the scenario is probably too easy. On the other hand, a player should not need to reload 15 times to get by a certain part of a scenario. That is frustrating and the scenario is probably way too difficult. The ideal scenario balance happens when a player gets stuck, but he knows that it's possible to complete the objective if only he did something a little differently. A player should not win by luck, the scenario should be constructed so that a player can learn from mistakes and use his skill to complete the objective.

One important item to note about scoring the balance category for scenarios where no fighting takes place, such as cut-scene scenarios, some puzzle scenarios and some rpg style scenarios, is that just because the player cannot die in such scenarios, that doesn't mean the scenario isn't balanced. You also need to take the author's original intent into account, giving the author some benefit of the doubt. If the author never intended the player to face a struggle to survive, then there's no reason to knock down the balance score if there isn't any fighting. So keep in mind that you do need to take the intent and goals of the scenario into account when scoring the balance category, especially for those scenarios where fighting is not included.

Multi-player scenarios are reviewed a bit differently in terms of balance. Each human player should start out in an equal position with equal starting resources and equal starting units. Obviously, the players don't have to match exactly, but they should be balanced. The map should also be examined to determine if all players have access to the same amounts of on-map resources. There are a lot of creative ways that map designers can use to make each player different, yet still balanced. If you choose to review multi-player scenarios, it's your job to ensure that each starting position is balanced with every other starting position.

Creativity

This area is probably second in subjectivity behind playability. Creativity is found in all aspects of a scenario, from trigger tricks, to map design, to the story, to what units a player is given, to the objectives, to sounds used, etc... Every aspect of a scenario factors into creativity. One thing to be careful for is not to knock points off of creativity if the designer uses a trick or anything else that you have seen used in another scenario. There's nothing wrong with using the same trick that someone else used and no reason to deduct points because of that.

Probably the biggest creativity factors are the starting position and the victory conditions. For example, any scenario that starts with a TC and three villagers with a conquest victory condition is simply not very creative. The farther a player gets from a random-style scenario, the better the creativity score. Keep all this in mind when rating creativity.

Map Design

Map design is one of the few categories that's very easy to define and give a rating to. I have pretty clear-cut rules on how map design is scored and this is how it should work.Map design is easier to define than the other categories, for it is often readily apparent whether a map looks good or not. With this in mind, some general (but not hard-and-fast) rules can be established. A random map is a 2. All a designer needs to do to score a 2 is to use a generated random map. Random maps can look fairly good, they function well and there's nothing wrong with using a random map in a scenario, but it's below average unless the random map is simply used as a base. If the scenario used a random map as a base, but proceeded to add details to it, then a score of 2 is obviously too low. There is nothing to stop a well-implemented random map scoring highly, but to score it as 1 would not fit. From that basis, it's easy to figure out where scores of 1, 2, 4 and 5 come from.

A rating of 1 is for a pathetic map... these usually consist of large blank areas with lots of square areas and straight lines. These maps look completely unrealistic and are quite unattractive. A rating of 2 is somewhere between a pathetic map and a random map.

A rating of 5 is for an outstanding map with lots of special details and concentrated effort to make the map much better than a random map could possibly provide. Obviously, a rating of 4 is given for maps that are slightly better than a random.

4s and 5s should be reserved for the best maps, ones that are easy on the eye, fit the purpose of the scenario, and are well implemented. It is up to the reviewer to decide whether a map is very good or exceptional, and rate accordingly.

One final note on scoring map design... only the portion of the map that can be seen during play should be scored. If there are large empty areas that a player never sees, that should not affect the map design rating.

Story/Instructions

This is another pretty clear-cut category. If there is no story or instructions, the score is easy... it's a 1. If there are instructions but no story, the max score is a 3. If there is any story at all, the rating goes up to a 4 and if the story is really good, the rating can be a 5.

This category is not as clear-cut as it first appears. Obviously a scenario without story or instructions would score a 1 (generally), but that does not mean that the automatic presence of either qualifies the scenario for a higher score. The presence of a decent story (not necessarily well-developed) and functional instructions should be the midpoint, a 3. Anything better or worse than these can be marked up or down accordingly.

If the instructions are wrong, misleading or confusing, the rating goes down. Also, keep in mind that in Age of Kings, the instructions and the story goes far beyond the pre-scenario instruction screen. Often the story is continued throughout the scenario by using trigger events to move the story along. Also, since objectives can change in the middle of a scenario, the quality of the instructions must be judged throughout the playing of the scenario.

Some other guidelines on scoring this category: An introductory bitmap is a nice touch and a good image can often raise the score, however, it is not required to score a 5. It certainly helps, but it is not an absolute requirement. Hints and History can also be judged here. These two areas are not required, but they can also help boost a scenario's score. While a bitmap, hints and history are not required, it would be difficult to give a rating of 5 if all three areas are missing. The rating should not be affected based on whether the story is fictional or historical. It doesn't make a difference as long as there's a story that draws the player into the scenario.

The last item that factors into the rating of the story and instructions is grammar and spelling. A designer should be diligent in this area of his scenario since it's very easy to copy the text into a word processor and spell check the instructions. There's no excuse for having spelling errors in a scenario... it simply shows a lack of effort on the part of the designer. The only exception I make is for designers whose primary language is not English, but many of our non-native English speaking visitors have excellent command of the language, so this is not a get-out clause.

I was in the process of doing something very similar andrew before I noticed you post. Some thoughts:

I very much agree with all your points. I do however, think that the category of Balance needs to be addressed for Cutscenes. As of now, cutscenes are getting disproportionately high scores, as they basically get a free 5 in this category (I know that according to the guidelines a perfect score can be lost on occasion, but for all practical applications this is always 5).

I think the balance score needs to be changed to a different category. My suggestion would be something like cinematic quality, or just to have 4 categories.


If this and the whole official reviewer thing gets added to your changes I for one would approve.

Just having 4 categories would be my 'realistic' ideal, getting rid of balance altogether and just counting it as part of playability (after all, can you really have a fun scenario that is completely imbalanced?), but it'll never happen. Not on this site. People are too set in their ways.

I'd also argue that Creativity is a white elephant of a category, being almost entirely subjective and abused. To make sense of the category, it has to have disclaimers along the lines of 'even scenarios that are not very original can score highly here', and that kind of undermines the whole category. If we were forced to keep categories, then I guess Playability, Map Design, and Story/Instructions would probably be my preferred option, although I do feel that 'Creativity' should be somehow represented in one of the others... probably merged with the other categories for various facets of it, really.

I would still rather get rid of categories altogether, but hey.

The bit about cutscenes is worth noting, anyway. That was an addition to the original review guidelines. There is surely no reason for amendments to be acceptable in 2001, but not in 2008.

[This message has been edited by Andrew Dunn (edited 06-30-2008 @ 02:34 PM).]