Research as just asking people

One of the strengths of ethnographic fieldwork is that anthropologists can observe what people really do, rather than what they say they do.  This is more difficult in urban environments, and takes longer, but it pays in data of greater reliability and value. Often, students think they can just go into a village or store and ask questions and get the answer.  I was reminded of this issue while listening to an NPR story about voter behavior research in the US.

All kinds of people have tried to divine the thoughts and feelings of the American voter. But until recently, the only way researchers and pols could figure to study a voter was to ask the voter questions. You either put them in a focus group or you polled them on the phone.

But according to Jennifer Green, another researcher who studies voters through experimentation, that's no way to study a voter. You have to use controlled experiments, she argues, because voters themselves often don't understand what moves them. Most of us, she says, don't.

"If I showed you a quacking duck and I said, 'Hey, do you think this would make you more likely to buy this insurance?' You would say, 'No!' You're going to say, 'I want to know how much it costs! What it will cover! All those details so I can make an informed decision.' We want to portray ourselves as people using information to make informed decisions."

But obviously, Green says, the things that move us often have nothing to do with what our conscious minds tell us is important. "The thing that makes an impression on us, changes our minds ... may be a quacking duck," she says. "And we only figure that out by testing. Asking people is not the same as testing."

This is a reference to the Aflac duck, which has been the mascot and main actor in many TV commercials selling insurance (here is one with the famous Yogi Bera, himself famous for witty sayings, though he also said "I really didn't  say everything I said."). This story points to how important it is to go beyond what people just say about their behavior. A questionnaire that asks "Will you vote?" does not test whether the respondent will vote, just whether they say they will vote. Too many students and researchers forget this. When I have pointed this out to academics (often in other disciplines), their only reply is, "Well, yes, but this is the best we can do."  No, that is not good enough. This story is an example of how much better the research is when we keep this distinction clear.