You must be logged in to post messages.
Please login or register

Scenario Design
Moderated by Yeebaagooon, nottud

Hop to:    
loginhomeregisterhelprules
Bottom
Topic Subject: The Download Section: A Review?
« Previous Page  1 2  Next Page »
posted 23 August 2008 01:03 PM EDT (US)   
As a long-standing member of AoMH, I feel it my duty to the community to speak my mind about the site, its systems and its community.

For a while, I have looked at the download section with disappointment. I feel the entire section needs a revamp. Whilst i know in terms of structure it's a lot harder to reorganise it from a moderation level, however, it's the systems such as the Best Files, Official Reviews, and general moderation which needs to be rethought.

Not to say that the moderators do a poor job, it's perfectly acceptable for them to miss poor reviews here and there as they've got plenty else on their plate. However, isn't that surely what the official review team is there for?

My comments regard mainly the campaign and single-player categories. Forbidden and Terminator Ace appear to do their specialist categories justice, however, Papaya for example, and no disrespect to him, hasn't posted a review since January 2007, and his last review of a campaign or scenario was of Emperor by Celsus on 2nd October 2006; empireofages has given just one review this year in his category of recorded games; Alex (Pepsidude) has given just two ratings of scenarios in the last year and none since April; Khan & Steak, the official reviewers' most prolific reviewer, has given just four ratings this year and none of them are single-player scenarios or campaigns...

What is possibly a cause of the inactivity of these reviewers is that they have moved on to bigger things or have reduced their activity on the forums. But is this good enough? This sort of thing should be kept track of, and the positions reassigned to more active and effective reviewers. The inactivity of these staff members means that more and more poor reviews are getting through the fish net, even when they are reported. To use recent reviews as an example, MasterBeef's Chronicles of Kamos has received a review on 13th July 2008 which is clearly in violation of the guidelines, yet it has gone unnoticed and uncensored. Even members have been amazed by this saying:

i'm surprised that none of the staff have made it to invade here
Other examples of poor reviews which have slipped through the censorship committee are Story_Master's, Sniper1988's #1, Sniper1988's #2 and eyecandy of doom's reviews.

What is even more shocking and frustrating for the hard working designer is that the top-rated scenario on the Download Section's Best Files database is there on a rating from an official reviewer and even that review falls short of what is acceptable as a decent review. These scenarios subsequently get hundreds of downloads whilst other scenarios are overlooked.

So what do I propose?

In essence the staff positions on the official review board need to be reassessed. There needs to be a standard that the official reviewers must adhere to in order to remain in their positions, such as quality of reviews and frequency of reviews. When non-staff members such as PurpleWorm, Baske and Banoffee appear frequently on the 'recent reviews' page having posted decent reviews it amazes me that this has gone on for so long.

What should they do?

Official reviewers should be required to review at least two files per month in their particular field (this number could vary slightly from category to category). These reviews should set a standard from which other member can follow. They should go above and beyond the guidelines. The three line rule is redundant. Just because somebody writes three lines doesn't means it's three lines of any substance, why should that review stay when a perhaps slightly more informed review that didn't post three lines in every category (although in my opinion you should post at least a decent paragraph per sub-rating) is deleted.

They should reassess the files on the Best Files page, check if the reviews still apply. Deleted, alter or re-review the files to reflect the modern perspective. Perhaps the official reviewers could dig up a monthly 'featured file' that they recommend users to download. I also believe every file in the 'Top-Rated' section should be rated official, too many times have I seen poor productions listed there.

How many should there be?

The single-player and campaign sections should have at least two official reviewers and perhaps at least one specific AOM Vanilla reviewer. Multiplayer should have at least two, recorded games at least one, mods and random map script at least one, and miscellaneous at least one. There should also be a head reviewer to 'watch the official reviewers' (as Guardian_112 once put it).

What about the Download Section itself?

The way the files are ranked is ineffective. Files which have been rated an average of 4.8 by 8 users for example should appear higher than files which have been rated 4.8 by one user. I would also say that file rated 4.8 by 8 users should appear higher than a 5 by one user.


I'm not attempting to undermine the authority here I just feel that this issue needs to be addressed and has needed to be addressed for a long time. Perhaps a Community-Staff Liaison Committee should be set up to improve site services and facilities and to improve communication between the two. I understand that this isn't a state, the internet isn't a democracy; however, what benefits us makes us happier and in turn benefits the site.

_____________________________________
When i said the three line system is redundant i didn't mean that it should be lowered. I believe the rule should be removed as forcing people to write three lines just to keep a substandard review is an ineffective system and makes it difficult for reviews to be deleted when they aren't really up to scratch.

As PurpleWorm says, one should be aiming at writing much more than three lines per sub-category but they shouldn't be thinking all the time "I must write at least three lines of rubbish or this will get deleted", that sort of mentality doesn't help. To quote PurpleWorm again, the review isn't only about saying whether or not you liked it, it's about suggesting improvements, alternatives, and other things that could be done to make the rating higher.

I also find it frustrating when a review for example gives a review like the following:

Playability: 4

Very enjoyable, its really fun to move around the map and complete the tasks. There is a great ambiance about the scenario. I really liked all the elements you included. It kept you playing right to the end


As no doubt you can see, a review with this text would have been kept as it fulfilled the 3-line rule, however, there is no substance to it. Great, so you enjoyed it... what did you enjoy, so far as the example review is concerned I can't tell if you've actually played the scenario at all. Why did you only give a 4? you've mentioned nothing negative so surely you should be giving a five here, no?

What needs to be done is to explain where the designer got each of the 5 points. You don't need to be pedantic about it (like saying you got your first point because...) but what would be a much better review would be the following:

Playability: 4

Immensely enjoyable to play. There were no bugs that affected the gameplay so I have no reason to take off marks there. However, I did feel that the missions got slightly repetitive by the end of the scenario. You managed to hold my attention throughout the scenario by including many 'life' factors which made the scenario feel realistic, such as the citizens of the town going about their daily business and you got bonus points for sending them back to their houses when it got dark, a good use of the day/night cycle here. Overall, a smoothly playable map albeit repetitive. Try and vary the objective to improve.


You can see here that the reviewer is stating exactly why they have rated it a 4, detailing any bugs, general enjoyment, extra features, etc and using examples to back up these comments. If you: 1) State how much it fulfilled the category; 2) state if their were any bugs or errors; 3) detail something you really enjoyed; 4) detail something you didn't think worked; and 5) summarize by suggestive how the rating can be improved, one can easily write 5 or 6 sentences within a couple of minutes. A decent review should take roughly 10 minutes to write (2 minutes per category).

So you can see here that there really IS 15 lines of information to write, if you check out some of my own reviews you can see that at no point am i talking about something else, unless you count improvement suggestions as going off-topic. Some of my reviews have had a word count of 700+, all of relevant information.

However, i concede, I am not expecting a sudden change. I am optimistic that as one of the oldest members still around these forums my views will be taken with a degree of respect. But it is the COMMUNITY that has to band together and force a change to happen.

ROME: 78 BC | | | | | THE LOST RUINS
Rome in your hands | Of Ancient Mesoamerica

______________________________

AOMh Member since 2004

[This message has been edited by AnnoDominius (edited 08-24-2008 @ 09:46 AM).]

Replies:
posted 23 August 2008 02:11 PM EDT (US)     1 / 29  
I agree with you 100%. And if they're looking for someone to review or moderate, I'm happy to take on the position. (although I rarely write more than 3 lines.) Not sure what I would normally have to do to get one of those positions, but as I said I'm available for them, and very active, I checked the site atleast 20-30 times yesterday cuz I was waiting for my taunt package to upload.

At any rate, I too have been lacking reviews, when I submit maps they go completely unnoticed. I strive to create only the most original, highest quality, and most fun maps. And I feel that I succeed in that. So why must I beg my friends if I want a review. (That doesnt even always work.)Hopefully this will change at some point. I review every once in awhile and hope other people will do the same, even tho they aren't obligated to. Oh and btw, I have an Idea to help out with the ratings.

Have 2 seperate ratings, official (by the official reviewers,) and public rating (by everyone else.) And drop the requirement to 2 lines (of substance) for the general public, and have the official reviewer still do 3 lines. That will encourage more people to review, sine there's less work involved. And hopefully the official reviewer will be trustworthy, so we can look at his category as the true rating, while the other one is a back up incase the official disagrees with the public.

-Die_Mortal

[This message has been edited by Die_Mortal (edited 08-23-2008 @ 02:18 PM).]

posted 23 August 2008 02:19 PM EDT (US)     2 / 29  
Although I agree on most of your points, specially imposing a minimum number of reviews per month, this is all wishful thinking.

Things here like to happen in slow motion. Mods that get promoted do get exicted and do one thing or another but soon the maintenance of this website returns to its normal rate. Sadly, I don't foresee any change at all on the top-ranked for example - this has been many times proposed before.
-invent00r

My Work: 1 2 3 4 5 6 7 8 9 10

WIP: Master XS Battle Micro AI
Paused: AI (%4.247)
Os segredos são de quem os souber guardar.
posted 23 August 2008 02:35 PM EDT (US)     3 / 29  
Have 2 seperate ratings, official (by the official reviewers,) and public rating (by everyone else.) And drop the requirement to 2 lines (of substance) for the general public, and have the official reviewer still do 3 lines. That will encourage more people to review, sine there's less work involved. And hopefully the official reviewer will be trustworthy, so we can look at his category as the true rating, while the other one is a back up incase the official disagrees with the public.
If any official reviewer ONLY gave 3 lines, I'd be upset. Anyone who is on the review team should be giving 15+ lines per category, and I expect any Average Joe reviewer to at least aim higher than they are now, 5+ lines at least. Reviews can't be reviews and be 3 lines. There's just not enough detail.

In fact, the length criteria per category is low imho, and should be raised.

I agree with AD's suggestions in entirety as well, but I suspect that they will fall on deaf ears...

PurpleWorm
Winner of the 2008 AoMH Boss Battle Competition
Scenarios: Knights of Ni, Gargarensis Rap (4.5)
"I think we consider too much the good luck of the early bird, and not enough the bad luck of the early worm." -Franklin D. Roosevelt

[This message has been edited by PurpleWorm (edited 08-23-2008 @ 02:36 PM).]

posted 23 August 2008 02:39 PM EDT (US)     4 / 29  
I said this months ago. Unfortunately, no one did anything about it.
posted 23 August 2008 04:06 PM EDT (US)     5 / 29  
You're living in a fantasy world then purpleworm. I've reviewed many times and found it difficult to write much more than 3 lines. The simple fact is, there ISNT anywhere near 15 lines of info to give, if someone writes a review that long, its because they are talking about something else, or are spewing nonsense in most of the review.
posted 23 August 2008 05:25 PM EDT (US)     6 / 29  
You're living in a fantasy world then purpleworm.
You just realized that? :P

I wouldn't call most of the longer reviews "spewing nonsense," after giving good constructive criticism (which always takes 5 lines for me at least), a good reviewer should move on to improving in the future, tips, and various comments directed toward both potential downlaoder and designer critiqueing in detail the aforementioned cconstructive criticism. If that doesn't take anyone 15 lines, I just don't think the reviewer is doing it right. Sure, some maps are going to require smaller or larger reviews, the point is that most reviews nowadays aren't of the utmost quality that I expect as a designer of a particular reviewer. A fantasy/idealist world maybe, but a well intended one. At the very least the minimum lines per category should be upped to 5 lines.

PurpleWorm
Winner of the 2008 AoMH Boss Battle Competition
Scenarios: Knights of Ni, Gargarensis Rap (4.5)
"I think we consider too much the good luck of the early bird, and not enough the bad luck of the early worm." -Franklin D. Roosevelt
posted 23 August 2008 06:12 PM EDT (US)     7 / 29  
When i said the three line system is redundant i didn't mean that it should be lowered. I believe the rule should be removed as forcing people to write three lines just to keep a substandard review is an ineffective system and makes it difficult for reviews to be deleted when they aren't really up to scratch.

As PurpleWorm says, one should be aiming at writing much more than three lines per sub-category but they shouldn't be thinking all the time "I must write at least three lines of rubbish or this will get deleted", that sort of mentality doesn't help. To quote PurpleWorm again, the review isn't only about saying whether or not you liked it, it's about suggesting improvements, alternatives, and other things that could be done to make the rating higher.

I also find it frustrating when a review for example gives a review like the following:

Playability: 4

Very enjoyable, its really fun to move around the map and complete the tasks. There is a great ambiance about the scenario. I really liked all the elements you included. It kept you playing right to the end


As no doubt you can see, a review with this text would have been kept as it fulfilled the 3-line rule, however, there is no substance to it. Great, so you enjoyed it... what did you enjoy, so far as the example review is concerned I can't tell if you've actually played the scenario at all. Why did you only give a 4? you've mentioned nothing negative so surely you should be giving a five here, no?

What needs to be done is to explain where the designer got each of the 5 points. You don't need to be pedantic about it (like saying you got your first point because...) but what would be a much better review would be the following:

Playability: 4

Immensely enjoyable to play. There were no bugs that affected the gameplay so I have no reason to take off marks there. However, I did feel that the missions got slightly repetitive by the end of the scenario. You managed to hold my attention throughout the scenario by including many 'life' factors which made the scenario feel realistic, such as the citizens of the town going about their daily business and you got bonus points for sending them back to their houses when it got dark, a good use of the day/night cycle here. Overall, a smoothly playable map albeit repetitive. Try and vary the objective to improve.


You can see here that the reviewer is stating exactly why they have rated it a 4, detailing any bugs, general enjoyment, extra features, etc and using examples to back up these comments. If you: 1) State how much it fulfilled the category; 2) state if their were any bugs or errors; 3) detail something you really enjoyed; 4) detail something you didn't think worked; and 5) summarize by suggestive how the rating can be improved, one can easily write 5 or 6 sentences within a couple of minutes. A decent review should take roughly 10 minutes to write (2 minutes per category).

So you can see here that there really IS 15 lines of information to write, if you check out some of my own reviews you can see that at no point am i talking about something else, unless you count improvement suggestions as going off-topic. Some of my reviews have had a word count of 700+, all of relevant information.

However, i concede, I am not expecting a sudden change. I am optimistic that as one of the oldest members still around these forums my views will be taken with a degree of respect. But it is the COMMUNITY that has to band together and force a change to happen.

ROME: 78 BC | | | | | THE LOST RUINS
Rome in your hands | Of Ancient Mesoamerica

______________________________

AOMh Member since 2004
posted 23 August 2008 06:14 PM EDT (US)     8 / 29  
Thank you for bringing this up, Pharoh. I agree with you 100%.

I know I have been neglecting my duties and I apologize. My only objection of giving up my position is that the files I've reviewed that have made it to the best files list will lose that rank. It looks like our review team has been pruned quite a bit already and we have lost several files from the best files list which deserve to be there. With that said, a whole revamp is easier said than done. We have been constantly looking for new reviewers - I believe we contacted and 'hired' two new guys earlier this month. And we have posted several ads welcoming reviewers to apply to join the team. I, myself, have encouraged reviewers to apply but still, we are not hearing from the number of reviewers we'd like.

So if you're interested in applying, we'd be more than happy to hear from you. Just send off an email to the Review Team Head, Khan and Steak, at dreamer@live.au. I'm not sure what exactly they're looking for but just write a short explanation of why you'd like to become an official review, link to one review which the team will examine, and be sure to include your original SN.

As for the mess of unmoderated reviews in the database, we (with the exception of Khan and Steak) have no more power over that than you guys do. I believe I Love Bananas is responsible for downloads administration. If you see a review that needs moderation, post the link here. I will try to do the same when I see poor reviews but cannot take any action.

A liaison committee will not be necessary. We want to hear your input but we welcome individuals to state their thoughts without having it looked over by a whole committee. We are approachable and you shouldn't hesitate to share your thoughts, preferably via email. You can email me at any time at alex@heavengames.com and I will address your concerns with you, the rest of the team, and the greater AoMH community, if necessary.

Any complaints about how the download section, itself, works, should be emailed to I Love Bananas. If he can help you out and fix any flaws, I'm sure he will. If not, he can forward your suggestions to Zen and Pecunia, who are in charge of the technical side of our download databases.

posted 23 August 2008 06:21 PM EDT (US)     9 / 29  
Thank you Alex, I appreciate you responding to this and would be more than happy to engage in email conversations on the subject. I can see, now you have explain, how resigning you position will cause difficulties in the Best Files section. The committee was a casual suggestion (mainly for dramatic effect) but i appreciate you assurance of approachability nonetheless.

ROME: 78 BC | | | | | THE LOST RUINS
Rome in your hands | Of Ancient Mesoamerica

______________________________

AOMh Member since 2004
posted 23 August 2008 06:43 PM EDT (US)     10 / 29  
And so you know, Khan and Steak are currently working on new and improved review guidelines. So far, they have written guidelines for cinematography and mp and sp scenarios. They're very detailed and use an easy to follow approach. If you have any suggestions, do feel free to post them in this thread.

I will be taking on the guidelines for modpacks, utility files, and recorded games and will be contacting the modding and replay community soon.

Khan and Steak and myself were both gone for a good part of the summer but we'll try to get things back up - whether that means going on a hiring spree, pruning, or finishing some of our original plans. Unfortunately, I'm now on a Mac so I won't be able to do much other than review vanilla files.

[This message has been edited by Alex (edited 08-23-2008 @ 06:49 PM).]

posted 23 August 2008 07:39 PM EDT (US)     11 / 29  
High five for Macs!
posted 23 August 2008 08:33 PM EDT (US)     12 / 29  
Ok, a few things, not that I have a real point to make:

First of all, to elaborate on "As for the mess of unmoderated reviews in the database, we (with the exception of Khan and Steak) have no more power over that than you guys do" as though it were unclear, Review Team members cannot edit out other reviews. We have to go through the same application BS. I think Yeeb, Khan and Steak, and ILB are the only people who can actually edit other reviews. Maybe RT members SHOULD get that power (I know I see a lot of craptastic reviews), but it would require a restructuring.


Secondly, I and Terminator_Ace were invited to the review team like... a few weeks ago. Since becoming an RT member, I wrote like one or two reviews. I don’t know that T_A wrote anything. After becoming part of the RT, it’s harder to write reviews.

I used to do BS like play half the map if it sucked and then go on a rant about how it sucked.

Now I feel like that would be unfair (it was always unfair, but nobody would have cared before I got "promoted" to RT), so I force myself to sit through the sludge. A review used to take me about 2-4 hours. Now it takes 6 or more. I had at one point a goal to review EVERY new Multiplayer map in the DL section. I think I reviewed like 10 or so, but half of those were just scathing reviews of maps that were unplayable for one reason or ten others.

Part of the problem with RT semi-inactivity is exactly what you're suggesting (not that your minimum line requirement suggestion is anywhere NEAR high enough to affect the RT), but RT reviews are already held at a much higher standard.

Sure, we knew what we were getting into (and I could have easily declined the invitation), but as an invitee I didn't feel that there would be much pressure to crank out a bunch of reviews each month. I thought that my main job would be to set an example for other people, so my duty would be to write a review here or there. Two a month would be great, one every two months would be acceptable, but below standards.


Thirdly, the three-lines per category rule (TLPC) is found in… nowhere that I could find. I don’t know why that became such a big thing, it’s not in King Jared’s post and it’s not in the Review Guidelines. I dunno, I think maybe it caught on as an unwritten rule, but it’s a very stupid rule. The rule exists as far as I know only to allow moderators to insta-delete the awful reviews without having to type up a huge summary statement about why the review was deleted. If you have multiple categories where you’re coming up short, your review most certainly sucks, and should be deleted on other grounds. Sometimes I think the answer is just “write more” but a lot of people (e.g. Die_Mortal in this thread) actually feel pressured by the TLPC rule, and that’s when the issue becomes more complicated.

The REAL problem with TLPC is when people are pressured by it and scrape out three lines through a rant although they say nothing of interest. It has become such a standards-cutoff where bad reviews that just get over are deemed ok because RT moderators basically forgot how to delete bad-but-long-enough reviews.

Length != quality, but moderators are now backed so deep into a corner about TLPC that they don’t delete long reviews at all anymore.

Instead of TLPC, each category should have two things: 1) A criticism (unless a 5 was scored, then they should have a detailed explanation about why the map was exceptional) and 2) a compliment.

That’s a MUCH better guideline because it gives direction to fill up content instead of just an instruction to write more. Although like it was mentioned before, review guidelines are getting a rewrite and I’ve already drafted that into the stuff that I’m rewriting, although I’m afraid it does have some problems (like scraping the minimum requirement for EVERY category would still be a bad review, but I don’t know how to write that in and I don’t want some asinine, bureaucratic rule like “reviews must contain 13 compliments and 10 criticisms).


To summarize (pretending I had a point), most maps NEVER get reviewed anymore. Like I said before, if you promoted more people to RT, you’d actually get less reviews. You might get a flurry of a few reviews when they first join, but equilibrium will set in. Official Reviews are much harder to write, so you’ll get fewer of them in the long run. The problem I feel isn’t so much the bad reviews getting through, but the low low numbers of reviews and the total lack of direction from moderators. “Write more, you must have at least 3 lines per category” just discourages people, turning reviews into a chore.

Not that writing reviews is that much fun anyway, but it should be about conveying information to the author about what they did wrong, how to improve, etc. and not so much about justifying the rating you end up giving.


This might be why you (Die_Mortal in this case, although in real life it’s a huge list including a younger Forbiddian) are having problems writing longer reviews. You should be thinking in terms of like “What did this author do wrong, what did they do differently than other people, how much did they simply recycle from other maps, what did I like about this map, what was boring or too difficult, did the triggers run efficiently and on-time, etc.” The problem also might be that you simply haven’t played enough other maps to know what the standards are or have a mental list of similar maps. If you play more maps and do more reviews, they get much easier. You’ll begin to see patterns in maps.

Your advertisement here!
posted 24 August 2008 00:34 AM EDT (US)     13 / 29  
Heya, thanks for your input on the download section. I myself have also touched upon all these problems around the staff forums, etc. You know the saying though, if you want something done right, you gotta do it yourself. So I'm working with Pecunia to roll out some improvements in the next year or so.

The major flaw with the official review system is that it only counts official reviews if the member is actively part of the review team. If a member of the team resigns, then all of their reviews are no longer counted as official. This, along with a system for reporting bad reviews (think: Youtube like) are the things we're working on.

Any way, hopefully way we can have a forum opened at HGM some day for download section feedback. In the mean time, you can email any download section suggestions or complaints to me (papaya@heavengames.com) and I'll pass them along.

Regarding the official reviewers reviewing... in the past the idea was mentioned so many times, so I finally started it. I got the gears in motion, but activity died down pretty soon. It's one of those things that everybody want's done, but nobody really wants to do. Hopefully it can be fully revived eventually.

EDIT: Also, just to clear up a few things, HG Staff basically works the same as the review team. When you first join you're psyched to do lots of work, do lots of changes, etc. After awhile, you start to slow down and get less and less done. That is partly the reason why things are so slow around here. All of us are volunteers. Everything you see done at HG, everything, was done by people volunteering their time. I'm not using it as an excuse, I'm just trying you give you guys a grasp of what goes on around here. I don't like it either, but I've become the same way.

oh you can wait for what I can give
you know what I am so you know how I live
try to look proud but you’re not in the slightest
its happening now and it’s always been like this.

[This message has been edited by PaPaya (edited 08-24-2008 @ 00:45 AM).]

posted 24 August 2008 02:31 AM EDT (US)     14 / 29  
Btw, just wanna let ya'll know that I wasnt in the best of moods when I posted that long reviews were nonsense, so I hereby withdraw that comment. Lol.

Edit: Alex, I sent 3 emails to Khan and Steak with my application to be a reviewer. every time it was returned saying it failed to be delivered. Is there anyone else I can contact or another way to contact him?

[This message has been edited by Die_Mortal (edited 08-24-2008 @ 03:15 AM).]

posted 24 August 2008 03:31 AM EDT (US)     15 / 29  
The flaw in the rating system here is the very reason why we developed the Mod Corps (we intend to open that up to scenario reviewing too), an organization dedicated to awarding helpful and complete reviews based on more specific categories that pinpoint the important aspects of a project. Although we haven't been as active as we'd have liked to have been, mainly due to a lack devoted reviewers, the concept of the group is to correctly place projects that have been unfairly rewarded or rejected in this system we've grown so fond of that we've taken no care to correct as the design community and its work develops in complexity.

That whole three-lines per category rule that Forbiddian brought up is absolutely sickening. It reminds me of an English class I had in 9th grade that forced you to have a certain number of sentences regarding a specific aspect of the subject in a very specific order. While this type of rule stands great as a method of developing better writing or as a guideline, it fails miserably when strictly enforced. It forces the individual to either expand or contract what they intended to say initially and ultimately limits the impact that the statement may have given to had the author not been limited by such a rule. A good reviewer may only have two sentences of comments in a particular category simply because there isn't much to say. In some cases, one sentence is plenty. It's the one and two word reviews that we're trying to prevent. Requiring three sentences for a category may lead the reviewer to begin to ramble, which is just as bad as not saying anything to begin with.

If this thread fails to achieve its objective, that being to make heavy modification to the inner workings of our file library and its review guidelines and methodology, I would like to welcome any of the posters in this debate to join our reviewing organization. I say this not as a sales pitch, but as a stand against bad reviews and bad reviewers alike, and an invitation for the people who obviously are concerned by our lack of project grading equality to help let designers know where they stand and help them improve.

The concept of becoming more comfortable and lazy in a position of responsibility that Papaya talked about is something that happens in all sorts of situations, even as a designer. Although I enjoy it, eventually everything seems more and more like an obligation, and you essentially train yourself to 'work the system', making it seem like you're doing whatever you can to improve whatever is in place while merely sliding by at the bare minimum. This is true in the volunteer world, the paid workplace, the government and even the military. All of the enlisted jarheads that are motivated to do anything and everything that their instructing sergeants command joining because they want to kill bad guys and be the best military personnel that a non-commissioned officer could ask for eventually settle in and loosen their gears, so to speak, as they become more comfortable with their environment. I think part of it goes hand-in-hand with the fear of responsibility. Everyone wants to look their best and make it look like their superiors made the right decision by putting them in their position, and make sure they don't look stupid in front of the people under them. On account of this mindset, people screw up because of the simple fact that we're all humans and we all make mistakes at one point or another. When the individual realizes that they've made a mistake and no one punishes them for it, they continue to make mistakes, often the same mistake. Eventually, they drift into a carefree frame of mind, while still aware of their responsibilities, less motivated to adhere to them.

I could go on, but I kinda need the sleep for now.
posted 24 August 2008 05:39 AM EDT (US)     16 / 29  
First of all of I would like to clear up some things.
What is even more shocking and frustrating for the hard working designer is that the top-rated scenario on the Download Section's Best Files database is there on a rating from an official reviewer and even that review falls short of what is acceptable as a decent review.
I became part of the review team shortly after (I think it was 2 days) I posted a comment of "Payday" on the 30/01/07 (01/30/07). Which means over 30 reviews that I had made where not up to Review Team standards and weren't expected to be. However after becoming part of the RT, people treat you as if you have always been in it, this is especially the case when you have just joined. People treat you differently and things just feel harder, as Forbiddian said:
After becoming part of the RT, it’s harder to write reviews.

I used to do BS like play half the map if it sucked and then go on a rant about how it sucked.
I never slagged anyone off just because there map sucked, as far as I can remember (prove me wrong?). However back then I had never heard of the 3 line rule and I couldn't care less if someone said my review was crap, because as far as I could see, people where happy just to get a review, let alone a good one. If people want me to go back and re-review every past map that I had reviewed then I am more than happy to. However that just means that for a long time I will be re-vamping my reviews instead of creating new ones, so if I do, please don't complain that your map didn't get reviewed.
When non-staff members such as PurpleWorm, Baske and Banoffee appear frequently on the 'recent reviews' page having posted decent reviews it amazes me that this has gone on for so long.
I don't want to come across as person that put others down, however Baske has posted a total of 2 reviews, the same goes for Purpleworm. Banoffee has posted 4, all with around the same length and the same or slightly better input than the Gates of Fire review that I made. Don't get me wrong I am well aware that the content of PurpleWorm's and Baske's reviews are of high quality however, even if these people where on the RT you would be hitting them for inactivity. I'm just guessing here, so if I'm wrong don't shoot me, but having a mod on the RT isn't exactly a good idea, how many reviews have you seen mods do?
There needs to be a standard that the official reviewers must adhere to in order to remain in their positions, such as quality of reviews and frequency of reviews.
I do agree with this 100%.
Official reviewers should be required to review at least two files per month in their particular field (this number could vary slightly from category to category).
I do not agree with this 100%. People don't get anything for being on the RT, that's why our turnover is so high. We do this in our spare time because we want to, not because we have to. We want to minimise reviewing becoming a "chore" as forbiddian put it. However I don't want to come across as promoting that 5 times a year is enough.
The single-player and campaign sections should have at least two official reviewers and perhaps at least one specific AOM Vanilla reviewer. Multiplayer should have at least two, recorded games at least one, mods and random map script at least one, and miscellaneous at least one. There should also be a head reviewer to 'watch the official reviewers' (as Guardian_112 once put it).
Unfortunately, there is just not enough will nor man power. Even if we had a huge recruiting, the same would happen as it does now, the first few week-months, great; the following months, not so great... tender resignation.
We're definitely not saying that we don't want anyone to join; as Alex said if anyone wants to join just e-mail us and we'll be happy as anything to investigate. As unlike most people both e-mail accounts are open and can be found on our profile.
The way the files are ranked is ineffective. Files which have been rated an average of 4.8 by 8 users for example should appear higher than files which have been rated 4.8 by one user. I would also say that file rated 4.8 by 8 users should appear higher than a 5 by one user.
This has been said many times, not just by forumers but also by previous and present members of the RT. I agree 100% with what you have stated, I only hope that something will be done this time (of course in due time).

I would like to thank you for brining this to our attention. I am also thankful that you do not want to bring about instant change, as change does take time, especially when lots of things are wanted to be changed. The RT will discuss this further ~ Khan

<||=================================||>
Proud Member of Forgotten Empires
96Reviews KaS Dota2 Halo 5 Reach
{ "Work until your idols become your rivals" }
༼ つ ◕_◕ ༽つ trGetUnitScenarioName(); ༼ つ ◕_◕ ༽つ

[This message has been edited by Steak And Khan (edited 08-24-2008 @ 05:49 AM).]

posted 24 August 2008 05:49 AM EDT (US)     17 / 29  
By the way, thanks for the input from everyone, especially AD for starting this thread.

I think everyone kinda had this feeling in the back of their mind (or more).

Your advertisement here!

[This message has been edited by Forbiddian (edited 08-24-2008 @ 05:49 AM).]

posted 24 August 2008 05:58 AM EDT (US)     18 / 29  
Anyone else see the pun in the title of this thread?

...
posted 24 August 2008 09:43 AM EDT (US)     19 / 29  
Let me respond to a few statements and questions raised in response to my initial thread.
Maybe RT members SHOULD get that power
I am fully in agreement with this, as far as I was aware I thought they DID get that power until I wrote this thread and discovered to the contrary. It seems counter-intuitive that a staff member whose job it is to write decent reviews and set the standard, can't delete those review which clearly don't meet it.
I wrote like one or two reviews. I don’t know that T_A wrote anything. After becoming part of the RT, it’s harder to write reviews.
You have written 12 reviews since 28th July and Terminator_Ace has written 6 since 2nd June, I really don't see why it is harder to write reviews once you're on the RT, indeed, your standard of reviewing should have been higher enough to begin with to have got on to the RT so no change to your review style is required surely.

And a review took you 2- 4 hours and now takes 6? I'm guessing you're including playing time which isn't really part of the equation. As i said earlier, in reality a review shouldn't necessarily take more than 10 minutes.

Part of the problem with RT semi-inactivity is exactly what you're suggesting (not that your minimum line requirement suggestion is anywhere NEAR high enough to affect the RT), but RT reviews are already held at a much higher standard.
I suggested that the 3-line 'unofficial' rule should be removed, you can't set a limit on what is a quality amount of text. It's the old saying 'quality over quantity', as long as the text is quality it shouldn't matter how long it is.
Like I said before, if you promoted more people to RT, you’d actually get less reviews... Official Reviews are much harder to write, so you’ll get fewer of them in the long run
I disagree completely. As i said above "your standard of reviewing should have been higher enough to begin with to have got on to the RT so no change to your review style is required surely". I don't think just because it review is official makes it any harder to right. As long as you're writing constructive comments it shouldn't make a difference. Take for example Alex's excellent official review, this is equally as good as PurpleWorm's unofficial review.
The flaw in the rating system here is the very reason why we developed the Mod Corps (we intend to open that up to scenario reviewing too)
Perhaps that's what's required. An unofficial review team that act as sort of 'competition' as you like for the official RT giving them an incentive to review more.
Which means over 30 reviews that I had made where not up to Review Team standards and weren't expected to be.
No, but surely the standard for the review team should be equal to that of the hoi polloi? A review by a member of the official RT is just something that sets a standard that other reviews should be achieving. And what's more, your past reviews are still deemed as 'official' in accordance with the Best Files section. If the example I had chosen of the Gates of Fire was a lower rating, for example a 4, and not quite up to scratch, then I would have been less condemning but that fact of the matter is that it is the TOP rated in the Best Files page and this surely undermines the value of such a list and causes other files to go unnoticed. I'm not saying you should go through a delete/modify all your old ratings, but surely any that have made it to the Best Files (not only your own) should be assessed for alteration or deletion?
I do not agree with this 100%. People don't get anything for being on the RT, that's why our turnover is so high. We do this in our spare time because we want to, not because we have to. We want to minimise reviewing becoming a "chore" as forbiddian put it. However I don't want to come across as promoting that 5 times a year is enough.
Naturally, everything we do here should be because we want to do it not because we feel obliged to do it. However, this differs slightly when you volunteer to provide a service. This is completely up to you but by accepting the position you are accepting the responsibility that comes with it. Just as you would be in the real world. Fair enough, you do not want a bureaucratic system of 'You must do 5 reviews this month or risk being dismissed' but there should at least be a gentle list of duties you are expected to live up to, if you don't do the suggested 2 rating per month one month it's fine, you may make up for it the next month, as long as your actions can be deemed acceptably fulfilling your duty.

I appreciate your response in this matter however Khan and it pleases me that those of position have taken this subject seriously.
By the way, thanks for the input from everyone, especially AD for starting this thread.

I think everyone kinda had this feeling in the back of their mind (or more).
It's good that AoMH are taking the communities comments on board. It is a pleasure to be an arbitrator of such communication.
Anyone else see the pun in the title of this thread?
Ah, Excellent. Somebody noticed! One of my more impressive thread names if I say so myself

ROME: 78 BC | | | | | THE LOST RUINS
Rome in your hands | Of Ancient Mesoamerica

______________________________

AOMh Member since 2004

[This message has been edited by AnnoDominius (edited 08-24-2008 @ 09:44 AM).]

posted 24 August 2008 10:03 AM EDT (US)     20 / 29  
My comments regard mainly the campaign and single-player categories. Forbidden and Terminator Ace appear to do their specialist categories justice, however, Papaya for example, and no disrespect to him, hasn't posted a review since January 2007, and his last review of a campaign or scenario was of Emperor by Celsus on 2nd October 2006; empireofages has given just one review this year in his category of recorded games; Alex (Pepsidude) has given just two ratings of scenarios in the last year and none since April; Khan & Steak, the official reviewers' most prolific reviewer, has given just four ratings this year and none of them are single-player scenarios or campaigns...
I noticed this too. That's why I suggested getting Forbiddan and Terminator_Ace on the team last month. I try to do reviews myself, but I know I haven't been keeping up to it. As you have mentioned other people, I will keep my eyes open for the next month.

With the bad reviews, I check the DL section almost every day (when possible) and get rid of the bad ones, but I admit I have had to question some of them.

I agree with pretty much all of your points (Especially the top files one). I will be emailing ILB to talk about reviving the downloads spotlight as the staff have wanted to do this for quite a while now.

Thankyou for your suggestions, you've made a lot of good points.

(Sorry for my vagueness, I've been out of the loop for 2 weeks.)

______________________________________ Yeebaagooon ______________________________________
____________________ AoMH Seraph ____________________
"You can't trust yeebaagooon to lead a rebelion, He would send everyone to steal mirrors so he could bask in his own brilliance." - Out Reach
"Yeebaagooon had never seen a more handsome man in all his life. He couldn't control himself, He needed to act. Gripping the mirror in his strong arms he kissed the figure before him..." - Out Reach
AoMH: Unfinished Scenarios|Singleplayer: Codename Ripto|Multiplayer: Minigames Z|CSC 7
Ex Seraphs Dictator, Spore Heaven Seraph

[This message has been edited by Yeebaagooon (edited 08-24-2008 @ 10:04 AM).]

posted 24 August 2008 10:20 AM EDT (US)     21 / 29  
Thanks Yeebaagooon, i'm glad i'm managed to achieve something good through this.

Also, just a question... Can I pay to advertise in your sig? I'm sure it'd boost my downloads... :-p

ROME: 78 BC | | | | | THE LOST RUINS
Rome in your hands | Of Ancient Mesoamerica

______________________________

AOMh Member since 2004

[This message has been edited by AnnoDominius (edited 08-24-2008 @ 10:20 AM).]

posted 24 August 2008 10:23 AM EDT (US)     22 / 29  
I think that there should be a second rating system which is not based on reviews but a simple click rating where the user simply selects how many stars to rate the resource as. That way people are more likely to rate resources so there should hopefully be less bias.

The boy with the mad imagination
Prepare for the ultimate duel!
Learn to use all my triggers and what you can do with them. Visit here.
Find out and download the transform trigger here.
Play some minigames I have created outside AOM including some 3D games here.
posted 24 August 2008 10:27 AM EDT (US)     23 / 29  
Thanks Yeebaagooon, i'm glad i'm managed to achieve something good through this.
I need a boot up, my brain has been overloaded with Italian ice cream for two weeks, so it's nice to get things moving again.
Also, just a question... Can I pay to advertise in your sig? I'm sure it'd boost my downloads... :-p
Well I live in Surrey too so you can ship the money across the roads of doom ;p. I was thinking of an advert agency, just give me a while to think of something decent to put in this sig. If not then yes, and if Yeebaagooon V2 fails, the quotes will go back in. *Dramatic chords*

______________________________________ Yeebaagooon ______________________________________
____________________ AoMH Seraph ____________________
"You can't trust yeebaagooon to lead a rebelion, He would send everyone to steal mirrors so he could bask in his own brilliance." - Out Reach
"Yeebaagooon had never seen a more handsome man in all his life. He couldn't control himself, He needed to act. Gripping the mirror in his strong arms he kissed the figure before him..." - Out Reach
AoMH: Unfinished Scenarios|Singleplayer: Codename Ripto|Multiplayer: Minigames Z|CSC 7
Ex Seraphs Dictator, Spore Heaven Seraph
posted 24 August 2008 10:56 AM EDT (US)     24 / 29  
Edit: Alex, I sent 3 emails to Khan and Steak with my application to be a reviewer. every time it was returned saying it failed to be delivered. Is there anyone else I can contact or another way to contact him?
Anyone experiencing this problem can email myself at alex@heavengames.com
It will all end up in the same place anyhow.

posted 24 August 2008 11:39 AM EDT (US)     25 / 29  
I think that scenarios(and perhaps all categories of downloads) should be grouped into a few classes like: amateur, intermediate and professional (but please give them good names). When I did my review, it was someone's RPG which was only partially done, but the demo they made finished off a neat chapter in their whole story. Other than a couple of nuisances, it was made well and was interesting. In terms of a demo scenario, I would have given it a 4-something, but in terms of every class of scenario, it was more like a 3. How could I give this partial scenario a big 4+ score like a well-made completed scenario?

I'm not sure how much can be done with this BBS software used for this site but I'm sure you could make new categories within single player and multiplayer scenarios, to keep from ranking the amateurs the same as the pros.

In all your science of the mind, seeking blind through flesh and bone
Find the blood inside this stone
Well, I know I've never shown what I feel, I've always known
I plan my vengeance on my own - and I was always alone
« Previous Page  1 2  Next Page »
Age of Mythology Heaven » Forums » Scenario Design » The Download Section: A Review?
Top
You must be logged in to post messages.
Please login or register
Hop to:    
Age of Mythology Heaven | HeavenGames