Insiders, Outsiders, and the Editing of Inconsistent Survey Data

Sociological Methods & Research, 2003

How should social science researchers deal with data inaccuracies? This article uses Web-based survey data collected from faculty members in three social science disciplines to document variation in views about data editing. Through an analysis of qualitative responses to a hypothetical vignette, the authors demonstrate that a wide range of opinion surrounds the "proper" use of data. Reactions are to some extent contingent on discipline and experience with different types of data and data collection methods. They also depend on characteristics of the data-editing situation-for example, whether the problem is with an independent or dependent variable. Even taking these social and situational factors into account, however, there is still substantial diversity in vignette responses. Normative standards that pervade other aspects of the research process have not yet emerged for data editing.

Quality & Quantity, 1995

It is widely accepted among survey researchers that the use of probes by their interviewers can lead to more intelligible, more complete, and more easily coded answers from respondents. Yet it is not hard to find examples of the use of probes in the survey research literature that have failed to produce demonstrably better data. Since the poor codability of respondents' answers is the product of a number of factors, merely training interviewers to be more skilful in their use of probes is unlikely to significantly improve the quality of survey data. If probes are to be used, they should be both standardized and used systematically. It would be better, however, to try to eliminate the need for interviewers to have to use probes. Two positive steps the researcher can take in this direction are: to define key terms properly, and to specify the response frameworks that respondents should adopt when formulating replies to particular questions.

Social communication involves a performer and an audience. What the performer thinks and what she says, is shaped by the social and cultural context of the communication situation. In every narrative, what has actually happened and what is said, is shaped or even deformed by this context. Oral history researchers usually are well aware of this. But what about quantitative researchers? I argue, that questionnaires can, in a way, be conceived as a special kind of oral history: Here, as in narrative interviews, the interviewed person tells the interviewer about her thoughts and past events. What the performer tells is also shaped by social and cultural context. Thus, questionnaires do not produce “harder facts” than narrative interviews. In the contrary, there are dangers. Using examples from survey data, I will discuss: 1) Who shapes the narrative? In ordinary narratives, the performer is the more active part, using cultural traditions to transmit the information she wants to tell. In...

Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 2018

Evaluating a long–term methodological norm – the use of interviewers who have no prior social relationship to respondents – we compare response patterns across levels of interviewer–respondent familiarity. We differentiate three distinct levels of interviewer–respondent familiarity, based on whether the interviewer is directly acquainted with the respondent or their family, acquainted with the research setting, or is a complete outsider. We also identify three mechanisms through which variability in interviewer–respondent familiarity can affect survey responses: the effort a respondent is willing to make; their level of trust in the interviewer; and interview–specific situational factors. Using data from a methodological experiment fielded in the Dominican Republic, we then gauge the effects of each of these on a range of behavioral and attitudinal questions. Empirical results suggest that respondents expend marginally more effort in answering questions posed by insider–interviewers, and that they also lie less to insider–interviewers. Differences in responses to “trust” questions also largely favor insider–interviewers. Overall, therefore, local interviewers, including those whom, in blatant violation of the stranger–interviewer norm, have a prior relationship with the respondent, collect superior data on some items. And on almost no item do they collect data that are measurably worse.

2013

The perils of insider research – what happens when the researcher 'goes native' interviewing within an institution 'analysing the linguistic or behavioural phenomena examined in terms of the internal structure or functional elements of a particular system' and presenting the results from an emic or insider's view. This paper examines some of the major issues that confront researchers who choose to enter the field. Dilemmas include: witnesses who were and are often still are, part of the cast of characters who contributed to the institution or organisation under review and observers who are acutely aware of the project and as such may be somewhat sceptical about the way in which they, their actions or their institution might be portrayed. Challenges for researchers identifying with the research setting are explored together with aspects of positionality, anonymity and confidentiality. Examples of insider research experiences from a range of studies, presenting bot...

1999

Introduction It is not unusual for survey organizations to conduct periodic expert reviews of ongoing survey programs; neither is it unusual to reach beyond the program staff and include behavioral scientists in such reviews. An expert review of this type would generally include an examination of the sample design, definitions of data elements, question wording, data collection procedures, and the criteria for publication. The question arises, however, are there other ways to conduct such a review? What if the focus of the evaluation were redirected? That is to say, what if attention was placed not solely on the substantive and methodological content of the surveys, but was also focused upon the various individuals and organizations who use the resulting data? What could be learned from such an endeavor? Would the information gathered from data users substantially add to the knowledge already obtained from the "typical" internal review conducted by content specialists, sur...

Proceedings of the …, 2012