These are a few subjective qualities of roleplaying games: factors to consider when choosing a game to play, or designing one from scratch.
Color
D&D: Fair
D&D was wonderfully evocative to those of us who discovered gaming for the first time in the '80s, but compared to all the creative worlds out there today, it's looking a little faded. The base game is a bland and generic Tolkeineque fantasy world, but there are some very cool published worlds. It's gotten a little better in 4th edition, with some less-typical races and distinctive specialties within each class.
A quick Google search will turn up loads of gaming systems, many available for free download. They differ from paid-for printed games in a number of respects, but mostly they're missing color. Not just literally, but figuratively: your average free RPG has bland prose, few or no illustrations, and if there is any background material it's sketchy, inconsistent, or just amateurish. This matters, because if the setting is not attractive, no one will play.
If you're going to create a new game system, invest in the creative side. Write some catchy scenes, find an artist with some talent, put some thought and style into your rulebook. Otherwise, few people will care to look at your dry black and white prose.
Mood
D&D: Heroic
D&D is a game of epic fantasy in the vein of Tolkein, who was himself strongly inspired by Beowulf and other northern European myths. Players are more than normal people, although they start out just a small cut above. Death is generally fairly easy to overcome, and magic ranges from uncommon to rampant.
Roleplaying is often an afterthought. When it does come up, it tends to be about killing the monsters and taking the treasure with style! This is not inevitable or universal, but the game's combat-oriented rules and long tradition of dungeon crawling encourage it.
A game should evoke an emotional response. Every game has a certain style. In some, death comes easily, with one wrong step. In others, the players are superhuman and nigh-invincible, even if they may look like normal people. Is there a titanic and overt battle going on between good and evil, as in The Lord of the Rings, or does evil lurk in the shadows while a brave few keep it at bay? Perhaps evil has already won, or perhaps the "heroes" themselves are evil or at least don't follow conventional morals (Vampire).
Different moods appeal to different people at different times. No game can cover them all, nor should they try, so consider what mood you want, and try not to let anything in the game contradicts this. At least, not jarringly.
Consistency
D&D: Very poor in the older editions, good in the newer
In 1st and 2nd edition AD&D, you had distinct mechanics for ability checks, proficiency checks, attacks, damage rolls, spell failure, magic resistance, saving throws, morale checks, initiative, and thief skills. There were even special mechanics for breaking down doors and bending steel bars (two separate mechanics). No joke.
By 3rd edition this was mostly cleaned up, but it's hampered by the old-school dice: six distinct dice, plus various combinations. When a newbie plays, be prepared to hear "what dice do I roll, again?" over and over.
When a game uses a single consistent mechanic throughout, or at least a small number of different mechanics, it makes the game easier to pick up. Even if this one mechanic is a little complicated, it's easier to learn one thing well than to memorize a bunch of different systems. When a game system is consistent, there's no hesitation. Ideally the player can make a roll while the DM looks up the difficulty, speeding things along.
An interesting trend in newer games, including The Dresden Files and A Song of Ice and Fire is to do away with the age-old distinction between attributes and skills: there's a single pool of skills that include things like raw strength. On the other hand, games that maintain the distinction can offer interesting interplay of attribute and skill; for example, in World of Darkness you can combine the Subterfuge skill with Presence to bluster past a guard, or Manipulation to fast-talk, or Composure to calmly bluff through.
Simplicity
D&D: Good
As a class-based and level-based game, D&D is pretty good in this respect. The older editions were quite simple, despite the many weird mechanics. The newer editions remain fairly simple, but the plethora of feats and prestige classes have made life more complicated. Still, at character creation, there are just six attributes, a few classes, a few races, a few skills, and a couple of feats to assign - less complicated than most games, though it still can be overwhelming for someone not familiar with roleplaying.
There are some really fascinating games out there that no one has played, or never seriously played, because they're just too complicated. A simple game is far easier for new players to pick up, and remains appealing when the groups just isn't up for much mental effort. Simpler games are also usually easier to extend or modify.
As with all the other elements of a good game, simplicity has to be balanced against other goals. Enhancing any of the others aspects of a game tends to cost in the form of more complexity, and that's OK. After all, the simplest game is flipping a coin, but it's not fun. Remember Einstein's great words: "make it as simple as possible, but no simpler."
Extensibility
D&D: Fair
It's easy to add equipment and spells, and even classes, and it's been done many times. Adding or converting D&D to other genres has not been encouraging, though. Does anyone actually play D20 Modern? On the other hand, D20 Star Wars works quite well, perhaps because the Star Wars setting is a fantasy universe at heart. The hitpoint mechanic fits the style of Star Wars perfectly, for example.
A good roleplaying game engine can handle a variety of settings, characters, and situations. You can't hope to include everything your players will want, so consider how easy it would be for them to add it themselves. Does the game use obscure formulae to calculate the characteristics of weapons? Does each class have a unique system of advancement, a complex tree of abilities? These might be cool features, and may be critical to what makes your game unique, BUT they'll make it harder for people to expand your game, and that's a missed opportunity.
As an example, the old World of Darkness game did not officially support mash-ups, but as we've seen in the past 15 years there's something about vampires versus werewolves that just appeals. The new edition was built in modules, with a core rulebook and a book for each "species" of supernatural, and they work together very well.
Flexibility
D&D: Fair
As always, the main determination of how much you can do in the game is the DM: he can always say "sure, make a... make a Strength check" or "nope, the rules don't say you can do that" (run away if you ever hear that last one).
Junior DMs are supported by a pretty comprehensive set of rules for non-combat, non-magical tasks, like lifting a boulder, swimming a moat, or fast-talking a guard, but they're typically obscure, and often use unique mechanics rather than the standard. This got better in 3rd edition, though 4th edition's unified system for non-combat challenges is disappointingly bland.
Now that voice chat is so common, the human element is much more fully present in online RPGs. That leaves one critical element missing: creativity. We've all had that moment playing a computer game where we've said "if I could just move the damn boxes..." or "why can't I try TALKING to the guards?" Computer games will never have the flexibility of tabletop roleplaying, and that's the critical advantage that pen and paper games must use.
A good game allows players to try anything, and has a system to help set the difficulty. The game designers can't anticipate everything, so this system will by necessity be a bit vague, but it should at least exist! Having some kind of outline gives the DM something to lean on, and a basis on which to defend whatever difficulty he sets.
As one example, an ideal game should have a good set of comprehensive skills. This is a topic in its own right, but the short version is that (in skill-based systems) it should be straightforward to determine the right skill for an unexpected task. This is generally easier if skills are broad.
Uniqueness of Characters
D&D: Terrible
D&D was absolutely terrible about this in early editions; fair in newer. Before feats, just about every 1st-level fighter was a carbon copy of every other, and even at level nine or ten there wasn't much to distinguish one non-spellcaster from another. Feats help enormously, at the cost of some simplicity.
About the only way one 1st-level fighter (or thief) differed from another was ability scores, which encouraged players to min-max. When all you've got going for you is your Strength bonus, it's going to be important to you!
No two people are alike, so for damn sure no two heroes should be either. A player's character is his personal avatar, and should reflect his wishes, which may be quite specific. A good system will let a player start quickly, with a largely random or unformed character, or take the time to add substantial unique detail. Ideally, neither character will be more powerful, just different.
As one small example, in many myths and stories a hero's choice of weapon was significant. Is there a material game difference between using a mace, a sword, or a spear? The former delivers a heavy blow even against an armored opponent, while swords are good for parrying, and against the lightly-armored, and a spear is an excellent defensive weapon, gives the advantage of reach, but is clumsy in close quarters. Perhaps it would be too much to try to simulate this in-game, but it's something to think about.
Randomness
D&D: Fair
The flat D20 mechanic makes it easy to calculate your chances. The lack of a bell curve means that nothing is ever more than 95% certain, or less than 5%, which means that one-in-a-million chances can and will happen.
There's a balance here: too much randomness, and the players will feel like helpless pawns, where the simplest thing can go wrong and they can count on nothing. Players like to have some idea of what they can do. But if there's too little randomness, they lose the sense of risk and unpredictability that's part of adventuring.
In general, this is well-served with a two-die system: rolling 2d10 or 2d6 gives a decent bell curve, where there are one-in-a-hundred outside possibilities, but most results will be within a few points of the mean.
Balance
D&D: Poor
D&D suffers from - indeed, basically created - the power-curve law of spellslingers and swordsmen. A 1st-level mage gets one stinking spell for the day, and then he's utterly useless, while a fighter can keep swinging until his (much higher) hitpoints are depleted. A few levels later, around 5th when mages get Fireball, things are more equal. Very soon after, any competent wizard can mop the floor with same-level nonmagical opponents, at least if he has a little time to prepare.
Aside from this, balance within combat is often poor. A high AC is nigh-insuperable for nonmagical characters. Weak fighters can contribute very little to a fight, with little to do every round except roll the die and report "I missed" over and over. 4th edition D&D specifically addresses this by giving each class useful abilities that can be used every round, such as a basic "zap" spell for casters.
If you want perfect balance in a game, play chess. In a roleplaying game, no two characters will be the same (or at least, they shouldn't!), and no two challenges will be either. That said, you still need to consider how each type of character and each set of options is likely to fare in common circumstances.
For example, it's assumed that a squishy wizard will not do well in a melee; but he should still be able to contribute, albeit at a discount. Likewise, a combat-oriented character should not be useless in social situations. Keep in mind both the balance amongst starting characters and experienced heroes; does each class grow at about the same rate?
Also consider the balance between characters and opponents. If PCs get too many advantages that aren't available to NPCs, you'll have to compensate with enemies that are obviously more powerful, or large quantities of lesser opponents.
These are a few roles that come up in most games; every type of character should have a useful role in each, though they'll only be the best at one or two:
- "Tank": straight-up melee combat, against a single tough opponent or a group
- Ranged combat
- Ambushes and skirmishes; mobile combat
- Support: buffing and healing party members
- "Mezzing": distracting, disabling, or turning the enemy
- Navigation and transportation: this includes maintaining the party's means of travel, whether that's grooming the horses, flying the spaceship, teleporting through the ley lines, or keeping on good relations with the vampire lords of the cities they pass through
- Investigation: getting information, collecting allies, finding weaknesses
- Social combat: intimidation, persuasion, subterfuge
To some extent, it falls on the players to be well-rounded. Someone who chooses a mentally flatlined thug has chosen to be useless outside of combat. That's his decision, and he must live with it. But the character creation process should discourage one-sided characters. For example, in Ash a character's social attributes let him or her purchase contacts and allies. This gives players an incentive to round out their characters; otherwise, social skills may seem useless, especially to hack-and-slash "munchkins."
Realism
D&D: Laughable
Realism is NOT the point of D&D. It's a heroic fantasy game, and the fact that a moderately powerful warrior can survive dozens of sword blows or a fall off a cliff reflects that. There are still jarring moments, such as the fact that said fighter will takes months to heal from near-death, compared to days for a 1st-level character.
Not all games are meant to be realistic; Exalted makes D&D look like Milennium's End. However, if a game system does keep the real world in mind, it will be possible to modify the system to be realistic, if that's ever a goal. It's much easier to take a realistic game and add fantastical elements than to rewrite a heroic game to reflect the real world.