After publishing a few reviews, I noticed that I haven’t really explained my basis for my RPG product ratings all that well. Consider this a manifesto and an explanation in equal measure.

I may seem critical in my reviews, but to paraphrase Tolstoy – the Good things are always more limited (and alas, easy to pass over in some cases) than the infinitude of possible errors and minute shades of Bad and/or just Not Good. I go in with teeth bared and blood in my eyes, but I never expect something to be perfect. I expect it to be average.

So.. what do I like in a game?
I like pulp that punches you in the balls and steals your Priceless Artifact for its Nazi Dinosaur employers.
I like Giant Robots that are only as good as their pilots, locked in strife, as Romance and Madness rage in the background.
I like stratified and vicious intrigue in the halls of power, where pulling a weapon means you’ve lost; where carefully-aimed scandal and boardroom sausage-making are the deadliest tools.

More germanely to this blog, I like Weird-ass fantasy.
Oz, Spencer, the fever-dreams of Dunsany and Coleridge, and my own ongoing folklore studies have influenced my tastes more than Tolkien or Lieber. Even Narnia is my kind of Weird, in its own way. Fey monsters with shortcuts to victory if you can only break the Magic that holds them in this world. Beasts that walk like Men (good, bad, hungry.. as long as they aren’t constantly going for “sexy” I’m down).
I like my hideous, twisted Evil beings to cackle madly and ride the Devil in the night. The cold, beautiful evils in the heights to sell their souls for power. The petty Evil of all men to turn away its eyes and damn itself with the easy path. I want the Good of a simple man to cut through it and blaze out over the world, even as his fellows turn, fall, fail, and die facing the corruption of the world. And I want every one of them as a potential player, not just the Heroes.
My games, unfortunately, have often been much more generic :/

I want my players to say, “Fuck you, I’m not going out on a mountain in a snowstorm” not because the cold will kill them, but in fear of the Yuki-onna and the Jotun; they must fear the forest not just for the HD of the monsters within, but for the half-seen things sniffing hungrily at their souls. (Links to follow as I post some of the more fucked-up monsters I’ve come up with in the last year..).
I’ve succeeded there on more than one occasion, up to the point where I had my players in near-hysterical paranoia in a sewer, up against 3HD worth of monsters.

I prefer my magic to be a rare and nifty double-edged blade rather than generic. Check the Archive posts to see what I mean. I like it when players seriously debate selling treasure, when they start keeping track of that sage three towns over who studies Dark Runes and spends his time collecting the inscribed Gem-Idols used by the Melniboniean emperors as currency.
Again, I’ve been dragged too often down the easy path, although keeping a commonplace book to jot down ideas I have, say, on the bus has helped greatly there. Damn you, ADD, I’ve probably lost more gaming material over the last 2 decades than I’ll ever publish.

I want my players to go off the rails, because they were a faint suggestion in the first place: I want them to MAKE a story, not “tell it with me”. I’m a referee, and hopefully always will be, not a Storyteller. I don’t want A Story for them to play out in what I buy. I want the web of relationships within a scenario explained, and hopefully some of the motivations behind them; I can riff off of a motivation a lot easier if I don’t have to infer it from reading the entire damned thing. Also, the players NEED to be able to fail meaningfully without necessarily destroying their entire adventure. I’m a mechanic: single points of failure (SPoF) – areas where failing stops everything cold – have no place in a working system constantly exposed to hostile action.

So here are my categories, along with a quick explanation of their rankings.
Utility
Could I use this on its own merits? Is the thing playable out of the box? File Formatting, Maps, and encounters all fold in, as does organization. Hiding critical encounter clues inside a monster stat-block three pages away from the first room they appear in is the tiniest bit counterproductive. If I can’t read your shitty hand-written and scanned prop, I can’t expect my players to do so. Language use is judged here on its function and clarity, not its artistic merits. If your writing is incoherent and ambiguous, I can’t adjucate with it.

1/10: Basically unusable. The file crashes my e-reader, pages are garbled, multiple critical parts of the content are left out. The author’s language leaves my eyes bleeding, or he makes retarded font choices. Also, the book could be severely disorganized, especially in PDF form. I have to generate craploads of content just to play (not to adapt, but to play it as intended). Maps and props are ugly and unpleasant to use (blueprint blues..). The book requires another supplement or secondary book to function (if you need a basic, “core” book you don’t get shafted here. Likewise, I consider the intent of the module: B1 is as much an exercise in learning to place encounters as it is a training module for the characters).
5/10: Readable, workmanlike, and functional. There are some annoyances, perhaps no index or Table of Contents, or page numbers are not marked. Some language errors, but no ambiguity in the actual rules text. Maps and handouts are readable, if occasionally hard to cross-reference.
10/10: Well-laid out. Finding information inside is a pleasure, and the writing is completely error-free. The maps are clear and may even include insets for particularly tricky or annoying bits. Chapters are coherent and arranged in a logical order. Files use hyperlink cross-references, and are well-bookmarked. All of the information I need to pick up and run is inside the covers.

Modularity:
I assume I’ll have to do at least some hacking with anything I buy – swapping out some of the encounters, changing out treasures, adapting backstories, whatever. This is a measure of how much work it will be for me to use it on the table, and how easy it will be for me to drop it in front of my players. Backstory (along with a bit of the encounters) folds into this as well. Canned encounters are a shortcut, but they make returning to the area difficult. You will likely have to excise and replace them if the players leave.

1/10: I’ll have to rewrite this completely, spending a week or more, to play with it. It could be impossible to put it into a campaign world other than the “intended” one.
B1 gets a pass on content, for example, as it’s an exercise in teaching a GM to key encounters. Dragonlance.. not so much. It may be awesome, but bludgeoning the “Awesome” into something other than railroad tracks takes an offensive amount of effort.
5/10: Requires some hacking, probably a couple day’s serious work and brainstorming. There are several elements that, even in a book from which I wouldn’t use everything, I want to steal.
10/10: Holy shit, I want to use this right now. Hacking is minimal, or assumed and accommodated in a non-insulting way – preferably with some simple suggestions. Whole chunks of the product can be lifted and dropped, even between genres, and they make me want to.

Weirdness/flavor, modularity, and utility are always in tension – you add detail to increase the flavor, and every one is a potential chain to the creator’s expectations about the game or to his own assumed world. I like Weird, but not everyone does.

Weirdness/Flavor:
Weirdness generally only applies to Fantasy; flavor is for other genres, and covers the amount of story – NOT STORYTELLING – that will help me get my head into the scenario. The baseline on this is pretty conservative – I expect most modules not to be especially Weird, although I judge modules set in a Weird ‘verse like Oz much more harshly if they ain’t. Weird is a measure of how many encounters are ones that screw with the players or their perceptions, and how much of an odd flavor there is to the occupants. Flavor, by comparison, is the amount of self-contained information presented. Mysteries (solved or not – I can run with either. How many of the things described are related to each other, and how intriguing or inspiring are the connections?
Lizardmen “who are at war with the Kobolds”? Bleah.
“Within the cave lies a trapped, transcendental visionary society of gaunt albino lizardmen. They worship the god of Hunger, and make sacrifices by eating and letting their god steal away their sustenance. They snatch eggs from other creatures to give their god the nascent life-energy within. Kobolds are particularly favored, but almost any young creature will do” Oh Hell yes.
Likewise, the ratio of flavorful magic to “yet another +1, ooh, there must be something with non-magic weapon resistance” shit.

1/10: Banal, pulp in the worst sense of the word. All encounters are straight fights, with only tactics to differentiate them. Any magic is taken straight from another product without embellishment, or offers only simple mechanical bonuses.
5/10: There’s the germ of a couple of interesting things, an interestingly odd monster, or a good framework on which to build. Something I see in it immediately inspires me to embellish the module in a creative way.
10/10: Ju-On.

Puzzles/Traps:
How much do the non-combat obstacles to the players hinder them? Can they be overcome with wit and ingenuity, or only with luck? How interesting are they? Are there rewards, or do the traps just fuck you over? While you might consider rumors to be just Weirdness/Flavor, they allow you to seed out-of-context clues for your players as well as contextual cues. And lie to them, of course, but enough have to be true that the PCs seriously consider every rumor they know.

1/10: No puzzles/traps, all combat. All of the puzzles and traps can be defeated with simple Skill rolls. The players are punished for being sensible when they’re not under time pressure, or a single trap can end the adventure instantly and is unavoidable. Few, if any, secret doors, or multiple secret doors are absolutely required to accomplish the central goal. Secret areas required to advance are hidden in nonsensical parts of the map, and can’t be found by means other than skill rolls or blunt “pass or lose your arm” choices with no clues whatsoever.
5/10: Some bland, some interesting, but the mechanism is sufficiently-explained that a reasonably ingenious character can get through. Most of the traps have at least one indicator to a cautious party, and all the puzzles have a clue available to the PCs in some way.
10/10: The non-combat sections are engaging. Secret doors open options to the party, rather than blocking all progress; when you die to a trap, it’s probably your fault. The party stands a significant chance of losing a resource other than HP or henchmen – like getting their maps fucked up, or risking an anti-magic field to open doors elsewhere, or even losing time. Riddles are always a plus. There are multiple clues to puzzles and traps, and they aren’t always right in front of your nose.
Note: The Tomb of Horrors rates about a 2 on this scale. Most of the traps are bullshit or non-sequitur, you have to progress through a completely linear series of challenges that test your Saving Throws more than your noggin, and the reward is getting boned in the arse (see what I did there?)

Character Engagement:
Your player, standing at metaphorical entrance to your module, will ask himself one question. “Why the fuck am I going in there when I could be getting ale and whores”. Money and power only motivate for so long. Revenge (legitimate revenge, not “haha, I killed your backstory, now go get the guy responsible”) is always good, but hard to integrate into the START of the thing. Treasures of virtue (You saved our children! You have our eternal gratitude! <3) are a great motivation for virtuous characters. But.. well.. yeah. Hen’s teeth. Mystery, especially potentially lethal ones, is always fun.
You have to tread a fine line here; herding PCs is like herding cats. A good module will have fish, wet food, a spot of catnip and a sumbeam at the end, and a fleabath looming behind.

1/10: All rewards are in GP, there is no reason for the characters to adventure other than “we are adventurers”, and nothing encourages them to remain but masochism and the desire to play the game. “So this old guy walks up to you and points you to the Adventurer’s Guild Help Wanted board, and..”. The adventure opens with a 5-page canned sequence and/or a script. Emotions are forced on the characters on a regular basis. Likewise, there may be a “triggered-cutscene” plot which motivates the characters only into jumping the rails.
5/10: Some non-monetary rewards, and if the players leave they might come back instead of moving on. Special encounters inside or outside the adventure area offer a goal with a reward other than “all the cash you can carry”. Even if it’s “you won’t get enslaved/executed”.
10/10: Your players asked to return, and don’t even quote Monty Python for upwards of 30 minutes at a time. You wind up looking at the clock at some point and going “Holy shit, when did that happen?”, and the only place left open for you to eat at is Denny’s.

Treasure Engagement
Bluntly, will the players feel wealthier at the end of the module, or just richer? Are they keeping treasure as savings for an equipment/character upgrade, or because they like it? How much of the non-magic treasure isn’t sacks of coins? Are they likely to use some or all of the treasure as a tool rather than sell it? Are the players going to have to – and choose to – go out of their way to get it out? B1 was always especially strong in this regard, with treasures ranging from old tapestries to a room full of preserved tools and a set of weightlifting gear.

1/10: All coins/generic jewels, all the time. Perhaps the designer of the module offloads all of the treasure generation onto a random table in another product, or leaves all the goodies in the open.
5/10: Several items are unique, and there are at least a few consumable resources for the party.  Jewelry is well-described if present, and there’s at least one themed hoard. The players face a choice of what to bring out, and there’s a reason to take one or more items of lower cash value.
10/10: Supplies and other items are scattered throughout. Some treasures have obscure worth, while others are valuable but bulky and fragile. Players must make (educated?) guesses about what to bring out, and there is a strong flavor to many or most of the items.

Aesthetics
This is subjective as all hell, of course. Your mileage will vary, but I will always explain why I like or dislike the aesthetics of a certain product. I tend towards surrealism.. but then my favorite artists range from woodcutters to Japanese pornographers to Pre-Raphaelites, and I can appreciate a well-crafted example of an art style I dislike. Here I judge the beauty of the author’s language and font choices, not their raw utility. The art should reflect on the text – it’s not enough to have a pretty picture, let me see and feel what I’m trying to make my players experience. If that be revulsion, grotesquerie and offense, so the fuck be it, but make me experience something.

1/10: May have no art at all. Comic Sans and Times New Roman throughout. The layout is messy, and its language infelicitous. Words may be used wrongly, or there is a glaringly non-sequitur picture inserted.
5/10: The author clearly put some thought into the layout. What art there is is related to the text, although it may be displaced. The art is decent, if not inspired, or there’s a particularly excellent choice somewhere within. Word choices show the author is aiming for a tone, and most of the sections of the book have consistent style(s).
10/10: Excellent art and language choices. The tone of the work is not only consistent, but evocative; the art not just technically excellent, but inspiring. Tricky or key set-pieces are illustrated, and well. The book looks good on your shelf and in your hands.

Overall:
An average of the other scores I’ve awarded, and a summary of my opinion.
It doesn’t have to score highly in all categories to get a good rating. One of my favorite gaming books is artistically atrocious and basically unusable as written, but tremendously inspiring and modular. And no, it’s not the 1e AD&D books, although that was a good guess. It’s a “systemless” book, and among the many sections are a compilation of monsters from folklore and medieval bestiaries, and a summary of herb lore and sorcerous devices. The rest of the book is trash, and I have many primary sources to draw from scattered around the house, but it’s lightweight and incredibly useful as a summary if I need something now. The same is true of the Judges’ Guild Ready Ref Sheets. I also have a few supplements for games I don’t even play. They have little to no “utility”, or are atrocities on the engagement front, but the flavor, inspiration, and aesthetic value keep them on my shelf.
Finally, I’m a cheapass. If I say I’m happy I bought something it means I not only consider it above-average, but felt it was worth money I could have spent on, say, steaks.