Everyone who is involved in anything thinks that they're an expert. They think they have a pulse on what the people want, simply because they've heard some people talk about stuff. And then they use this information to guide judgement.
So here's your public service announcement, followed by a wall of text!
Friends don't let friends use anecdotal evidence to make decisions on how a game should operate.
So when running a game, there's really two different types of decisions you can make, and neither of these decisions actually benefits from anecdotal info.
The first types of decisions are strictly game design. When I say game design, I mean basics of a system, balancing, mechanics, levels of complexity, and so on. Anecdotal evidence is the enemy here for two reasons. Not everyone is wired for game design. There are a lot of nuances to how things should be put together. Asking for input from people who are untrained, or even worse, not shown the big picture is asking for trouble. And even then, people are very resistant to change, especially if it seems like they're losing something or if someone else is gaining something superior.
Don't get me wrong, feedback isn't bad. But, for the most part, this isn't a give and take. You're looking at the big picture. If people come back and may not be thrilled with your idea and their reasoning doesn't really match the goal of the change or the system in general, then you're in the clear. However, if they show you that you don't achieve your goal, find loopholes, or show you that everyone hates it, then you react in accordance with them. And this requires looking at all the responses as a whole, and that means having that stuff written down. Word of mouth or a story that one guy said is not going to cut it. I straight up tell anyone who gives me word of mouth feedback to mail it in or post it up somewhere.
The second kinds of decisions are feedback on enjoyment of the game. Unlike game design, this is less predictable and less structured. You may run two games the same way in two different parts of the country, and they could diverge greatly, simply because players in different regions may play for wildly different reasons. On top of that, gamers tend to be melodramatic.
I'm going to catch hell for this, but my wife does this all the time. Anytime anyone increases the price to a game or to Origins, or changes service, her immediate response is "I'm never going back, they lost my business." But that's normal for a lot of people. When first approached about an idea, the first reaction is to push back, no matter how illogical it might be. Unfortunately, a lot of anecdotal evidence is based on this initial push back.
My rule of thumb is that I get opinions on this in writing. This does two things. First, anyone who does not have a strong opinion won't bother writing anything. If you don't spend your time to write something down, then it probably wasn't enough of a deal for you to quit or even change your habits over. The second reason is that people will usually take a moment to think about their opinions when you're getting official information, particularly when it's public. That means that the inevitable initial push back won't cloud up the data.
This isn't directed at anyone in particular. I'm on rules staff at both WAR and for NERO National, and let me tell you that lots of people are guilty of using anecdotal evidence. Even I do it sometimes.
But that doesn't mean you should tolerate it.
In general, I totally agree! Particularly, when it comes to situations that can broadly be described as "people knowingly being jerkfaces." The fact that you know someone who's awful doesn't invalidate an honor-based game system.
ReplyDeleteOn the other hand, there are a small portion of cases where it's legitimate for a player to want to tell you something verbally rather than writing it down:
-It's a potentially emotional/volatile issue, and they are worried about their tone coming across wrong.
-They're terrible at expressing themselves in writing, and it's a complex issue.
-It's a personal issue, and they would rather share the information in a way that can't be easily seen by others.
I say this from the standpoint of a person who puts everything in writing, and occasionally regrets it!
But in terms of general rules/system/game griping, yes writing is a jillion times better, because it's easy to reference back and evaluate things cumulatively. And anything that gets people to sit down and think before they talk is great. Over at the Forge forums, Ron Edwards used to deal with volatile threads and pushback by sometimes putting "waiting periods" on replies. He'd address whatever they said, then ask that they read what he wrote, wait until the next day, and then respond. I always thought it was rather clever.
Re: "I'm going to catch hell for this"
Not cool, man. Maybe pick an example of this that's your own behavior, or someone whose permission you've asked? At the least, don't name names ;)
I always consider anectdotal evidence to be of minimal value because the sample size is nearly always 1. Testing with a large group with definite measurements is more valuable than even written feedback, in my opinion.
ReplyDeleteI agree with all of this, but I'd like to echo what Jyn said. In the case that it's a personal/emotional issue-- it may strike that one person as really important, even though it doesn't effect the rest of the game. For instance, if someone was really upset by a certain mod because "it might scare people (and by people I mean me), and I know this because someone told me... in fact, many people have told me so," then this sort of suggestion might bear consideration. But, yeah, in terms of rules and stuff, I think there needs to be, as Tim said, a larger sample size.
ReplyDeleteOn game design, I feel you're advancing two arguments: one that is technocratic (the idea that those with the most "training" and information should contribute to the discussion, and thus, that feedback from those outside of the "inner circle"/game staff, should be ignored) and a second that is simply elitist (i.e. those you essentialistically describe as "wired for" game design should be the only people whose feedback is considered). Both these arguments are, for me, "dead on arrival". When last I checked, there is no academy of game design churning out rules committee persons, nor is there any correlation between degrees in macro-economics and people writing LARP rules. Any training or education people receive in designing rules or running a game event is informal, experiential, and (the most subjective of some very subjective things) based on talent. It is itself a sort of anecdote (in the sense that it is a "short, amusing story", i.e. a fiction) to propose that only those with a non-existent qualification have their feedback considered.
ReplyDeleteI think you're likewise terribly abusing the term "anecdotal" when really you simply mean "oral". Nothing about getting feedback in writing makes it any less anecdotal than oral feedback, assuming that the person taking the feedback has the discipline to document the oral feedback and form a written record. Nonetheless, I have some sympathy for the gamerunner who is bombarded with verbal feedback: who wants to be playing transcriptionist all the the time, especially in the context of finding your latest rules change/plot idea/game event wasn't well-received?
That said, I feel like this whole post could be titled "in defense of ignoring feedback", and I really caution you to indulge yourself in such arguments. Think about the logical application of these principals to a certain national LARP franchise corporation owner. If you're wanting him (or others, or whoever/whatever inspired this) to use more empirical methods (and abandon all that anecdotal, or really, oral evidence), you'd be advised to talk about the methods you're suggesting rather than simply posting this polemic against feedback.
Actually talking about survey methodology would be a great topic for a post, e.g. instituting a formal survey (if your game doesn't already have one; NERO is the only system I have heard of doing this, and I think it is a strong point), and putting quantitative measures on it (I think NERO does this?), as well as guidance on breaking the survey down into component (or even atomic, in the information science sense) would make for a more worthwhile article.
@Jyn and Zoe
ReplyDeleteYou're right. There are certain times that vocal information is important, especially in stressful or highly emotional situations. Ideally, neither game design or general game feedback would fall into that category. Those situations usually require mediation.
@A
I'll agree that I did use anecdotal vaguely and that written evidence can also be anecdotal (but at least can be easily refuted and thrown out).
As to game design, I think you completely misconstrued the point. The people who own the game write the rules. If anyone doesn't like that, they can deal with that. The problem is that a lot of players give feedback, and if their suggestions are not met to the T, they assume they're being ignored.
I don't advise anyone to ignore anything (other than "someone said that x would ruin the game" without any legitimate information). But often players who share the same goals as the ownership and have proven that they make logical decisions and can be swayed by logical arguments are put onto rules groups. Players who don't share the same goals for the game are not the kinds of people you want in charge of rules.
Saying that some people are not as good as others at something is not elitist. I'm not saying that specific groups are the only ones to do rules design. I'm saying logically minded people are. And while feedback is highly appreciated, players shouldn't feel entitled to get what they want. That certainly isn't how it's worked with any major MMO.
Really, I should have called this post "filtering feedback" instead of anecdotal evidence. I'm not to worried. It's not the first time my goal shifted in the middle of the post. But I certainly do not subscribe to (or suggest in any way) that feedback be ignored completely.
Finally, there are many different ways that feedback is gained in many games, and I would say that NERO is one of the worst for this (but since most feedback is on the local level, YMMV). While they tend to take a lot of surveys, very little comes from that collected data. If you're really interested in a successful means of collecting feedback from players, I suggest you do some research on Accelerant Games and the PEL (Post-Event Letter) that they use. It really is a much better form of feedback.