In this paper, we show that in Web questionnaires verbal and visual languages can be used to crea... more In this paper, we show that in Web questionnaires verbal and visual languages can be used to create groups and subgroups of information, which influence how respondents process Web questionnaires. Following Schwarz (1996; and also Schwarz, Grayson, & Knäuper, 1998) we argue that respondents act as cooperative communicators who use formal features of the questionnaire to help guide them through the survey conversation. Using data from three Web surveys of random samples of Washington State University undergraduates, we found that when response options are placed in close graphical proximity to each other and separated from other options, respondents perceive visual subgroups of the categories, increasing the likelihood that they select an answer from each subgroup. We also found that graphical proximity creates subgroups with and without the use of category heading to describe the subgroups and that the addition of a verbal instruction to "please select the best answer" encouraged respondents to select one answer from each subgroup instead of overriding the effects of proximity. In addition, the effects of grouping were consistent when the subgroups were positioned either vertically or horizontally in relation to each other. Lastly, we found that the effects of visual grouping are consistent across both opinion-and behavior/fact-based questions, although the effects appear to be greater on opinion-based questions. Our findings contribute to the increasing evidence that both verbal and visual languages influence how respondents process and respond to surveys. Because respondents interpret the verbal and graphical features of survey questionnaires as relevant to answering the survey, inadvertent or stylistic design changes can influence how respondents process and respond to survey questions.
We utilize and apply visual design theory to experimentally test ways to improve the likelihood t... more We utilize and apply visual design theory to experimentally test ways to improve the likelihood that web respondents report date answers in a particular format desired by the researcher, thus reducing possible deleterious effects of error messages or requests for corrections. These experiments were embedded in a series of web surveys of random samples of university students. We seek to examine the sequential and cumulative effects of visually manipulating the size and proximity of the answer spaces, the use of symbols instead of words, the verbal language of the question stem, and the graphical location of the symbolic instruction. Our results show that the successive series of visual language manipulations improve respondents' use of the desired format (two digits for the month and four digits for the year) from 45 percent to 96 percent. These results suggest that writing effective questions for web surveys may depend as much or more on the presentation of the answer categories/ spaces as the question wording itself.
In this overview, we discuss the current state of survey methodology in a form that is useful and... more In this overview, we discuss the current state of survey methodology in a form that is useful and informative to a general social science audience. The article covers existing challenges, dilemmas, and opportunities for survey researchers and social scientists. We draw on the most current research to articulate our points; however, we also speculate on both technological and cultural changes that currently influence or may soon affect the efficacy of different methodological choices.
Researchers who are interested in small towns and rural communities in the United States often fi... more Researchers who are interested in small towns and rural communities in the United States often find that they need to conduct their own sample surveys because many large national surveys, such as the American Community Survey, do not collect enough representative responses to make precise estimates. In collecting their own survey data, researchers face a number of challenges, such as sampling and coverage limitations. This article summarizes those challenges and tests mail and Internet methodologies for collecting data in small towns and rural communities using the U.S. Postal Service's Delivery Sequence File as a sample frame. Findings indicate that the Delivery Sequence File can be used to sample households in rural locations by sending them invitations via postal mail to respond to either paper-and-pencil or Internet surveys. Although the mail methodology is quite successful, the results for the Internet suggest that Web surveys alone exclude potentially important segments of the population of small towns and rural communities. However, Web surveys supplemented with postal questionnaires produce results quite similar to those of mail-only surveys, representing a possible cost savings for researchers who have access to Web survey capabilities.
The expertise of behavioral survey methodologists is needed in national statistical agencies beca... more The expertise of behavioral survey methodologists is needed in national statistical agencies because of the necessity of using theory and research from the social sciences to reduce survey error. In this paper various social science based explanations for measurement error and nonresponse error are described in order to illustrate the conceptual foundations of such error reduction efforts. Three roles for behavioral survey methodologists in national statistical agencies are then discussed. They include: 1) bring an error reduction perspective to bear in an influential way for all aspects of designing and implementing agency surveys; 2) bring theoretical efficiency and effectiveness to experimental tests of alternative questionnaire designs and implementation procedures through the use of theories, concepts and pretests and findings of past behavioral science research; and 3) contribute to the expanding science of survey methodology.
Previous literature has shown the effects of question wording or visual design on the data provid... more Previous literature has shown the effects of question wording or visual design on the data provided by respondents. However, few articles have been published that link the effects of question wording and visual design to the development of questionnaire design guidelines. This article proposes specific guidelines for the design of establishment surveys within statistical agencies based on theories regarding communication and visual perception, experimental research on question wording and visual design, and findings from cognitive interviews with establishment survey respondents. The guidelines are applicable to both paper and electronic instruments, and cover such topics as the phrasing of questions, the use of space, the placement and wording of instructions, the design of answer spaces, and matrices.
This paper reports results from a series of experimental manipulations of check-all-thatapply que... more This paper reports results from a series of experimental manipulations of check-all-thatapply questions in an Internet survey. One purpose of the experiments was to determine whether reversing the order of the presentation of response options resulted in order effects, including primacy and anchoring A second purpose was to determine the effects of converting check-allthat-apply questions to a forced choice-format (e.g. Yes/No for each item). We found that the existence of order effects in check-all-that-apply questions appears to depend on whether questions require respondents to use temporarily or chronically accessible information, as has been previously reported for items requiring selection of only one response option. The conversion to a forced/choice format increased the percentage of respondents answering affirmatively to each response option. Additionally, forced-choice formatted questions were unaffected by the use of more active answer categories (e.g. fan/not a fan) as opposed to the common yes/no format. Finally, the results from this survey of a random sample of 1503 University students, were quite similar to results from a previous mail survey experiment, suggesting that the response patterns observed in the current experiment result from selfadministration in general, and not from a unique characteristic of web surveys. Respondents to self-administered surveys are often asked to answer a question by checking only those response options that apply to them. For example, they might be instructed, "Please indicate which of the following sources of information you have used to find employment in the last month by checking all that apply." Each information source in the list would then be marked only if it had been used.
Recent experimental research has shown that respondents to forced-choice questions endorse signif... more Recent experimental research has shown that respondents to forced-choice questions endorse significantly more options than respondents to check-all questions. This research has challenged the common assumption that these two question formats can be used interchangeably but has been limited to comparisons within a single survey mode. In this paper we use data from a 2004 random sample survey of university students to compare the forced-choice and check-all question formats across web self-administered and telephone interviewer-administered surveys as they are commonly used in survey practice. We find that the within-mode question format effects revealed by previous research and reaffirmed in the current study appear to persist across modes as well; the telephone forced-choice format produces higher endorsement than the web check-all format. These results provide further support for the argument that the check-all and forced-choice question formats do not produce comparable results and are not interchangeable formats. Additional comparisons show that the forced-choice format performs similarly across telephone and web modes.
How to cite Complete issue More information about this article Journal's homepage in redalyc.org ... more How to cite Complete issue More information about this article Journal's homepage in redalyc.org Scientific Information System Network of Scientific Journals from Latin America, the Caribbean, Spain and Portugal Non-profit academic project, developed under the open access initiative
Antonia Aelterman Se-yuen Mak John Ainley Carmel McNaught Sonia Ben Jafaar Judith Meece Anne Boom... more Antonia Aelterman Se-yuen Mak John Ainley Carmel McNaught Sonia Ben Jafaar Judith Meece Anne Boomsma Martina R. M. Meelissen Roel Bosker R. Daniel Muijs Adriana G. Bus Rebecca Norman Derek S. P. Cheung Marie-Christine Opdenakker Geraldine Clarebout Constantinos Papanastasiou Casey Cobb Thea T. D. Peetsma Peter Daly Hans Pelgrum Jeffrey Day Ellen Reames Cees M. de Glopper Jill Salisbury-Glennon Elisabeth De Groot Shifra Schonmann Bernadet de Jager David M. Shannon Perry den Brok Charles Teddlie Don Dillman Kirsi Tirri Harrie M. C. Eijkelhof Joke Torbeyns Barry J. Fraser Kwok Tung Tsui Peter A. Frensch Jan Van Damme Femke Geijsel Eva Van de gaer John Gray Greetje P. C. van der Werf Kathy Green Karen Van Petegem Henk Guldemond Peter Van Petegem Jan-Eric Gustafsson Marcel Veenman Philip Hallinger Jean Pierre H. C. Verhaeghe Egbert G. Harskamp Ludo Verhoeven Michelle Helms Peter P. J. L. Verkoeijen Lynda Idleman Nico Verloop David Kember J. D. H. M. Vermunt Paul A. Kirschner Adrie J. Visscher Meta L. Krüger Helen M. G. Watt Leonidas Kyriakides Felicity Wikeley Yuen-kuang Cliff Liao Theo Wubbels Hans Luyten Educational Research and Evaluation Vol. 12, No. 1, February 2006, pp. 95
Survey methodology now faces an unprecedented challenge for how to collect information from sampl... more Survey methodology now faces an unprecedented challenge for how to collect information from samples of people that will provide scientifically defensible estimates of the characteristics of the population they represent. Many decades of research have shown that in order for such estimates to be made with known precision four major sources of error-coverage, sampling, measurement and nonresponse-must be controlled (Groves, 1989). Subsequent research has produced a great amount of knowledge on how those sample estimates are affected by different survey modes, sample sources, sample sizes, the failure of certain types of people to respond to survey requests, and how questions are structured and worded.Today's challenge stems from many considerations. Response rates for some survey modes, especially voice telephone in national surveys, have fallen precipitously and are not expected to recover (Dillman, Forthcoming). In addition, RDD landline surveys miss nearly half of all household...
These are challenging times for survey methodologists. The telephone, which once seemed the heir ... more These are challenging times for survey methodologists. The telephone, which once seemed the heir apparent to face-to-face interviews, is no longer an obvious choice for conducting many surveys. It is being adversely affected by the change in connectedness of telephone instruments from households to individuals and a cultural shift whereby people are able to control the telephone rather than vice versa, as was the case during its rapid rise to prominence as a mode of data collection in the late 1970s to early 1980s (Dillman 1999). The internet has not served as an immediate source of salvation. It is limited by restrictions on access, an inability to develop sample frames, and response rates that are mostly lower than those achievable by telephone (Shonlau, Fricker, and Elliott 2002). Other data-collection methodologies, from Interactive Voice Response (IVR) to classroom and home administration on computers, and mixed-mode designs are being proposed as unique solutions for specific survey topics and populations. Random sampling from defined populations is being forgone by some, replaced by modeling of data collected from volunteer panels. The current situation, seen from a methodological perspective, has elements of being both the best and worst of times. Less than a decade into the era of electronic surveying is not a time for either despair or jubilation, but I think it is an appropriate time to take a look at where survey methodology has been and where it might be going. Specifically, I will talk about changes in our uses of survey modes, new areas of needed research that result from those changes, and some implications for AAPOR. The context for these observations is set by perspectives gained over the course of my career with a variety of survey modes. These experiences don a. dillman is the Thomas S.
ñolas-como Cataluña y Euskadi-el sistema comunicativo, y también el sistema político, es muy dife... more ñolas-como Cataluña y Euskadi-el sistema comunicativo, y también el sistema político, es muy diferente a los sistemas que se han configurado en la capital del Estado, que, por cierto, tiene unas características muy singulares.
Journal of Continuing Education in the Health Professions, 2010
... Login to save citations to My List. Citation. Database: PsycINFO. [Book; Authored Book]. Inte... more ... Login to save citations to My List. Citation. Database: PsycINFO. [Book; Authored Book]. Internet, mail, and mixed-mode surveys: The tailored design method. (3rd ed). Dillman, Don A.; Smyth, Jolene D.; Christian, Leah Melani. Hoboken, NJ, US: John Wiley & Sons Inc. (2009). ...
In this paper, we show that in Web questionnaires verbal and visual languages can be used to crea... more In this paper, we show that in Web questionnaires verbal and visual languages can be used to create groups and subgroups of information, which influence how respondents process Web questionnaires. Following Schwarz (1996; and also Schwarz, Grayson, & Knäuper, 1998) we argue that respondents act as cooperative communicators who use formal features of the questionnaire to help guide them through the survey conversation. Using data from three Web surveys of random samples of Washington State University undergraduates, we found that when response options are placed in close graphical proximity to each other and separated from other options, respondents perceive visual subgroups of the categories, increasing the likelihood that they select an answer from each subgroup. We also found that graphical proximity creates subgroups with and without the use of category heading to describe the subgroups and that the addition of a verbal instruction to "please select the best answer" encouraged respondents to select one answer from each subgroup instead of overriding the effects of proximity. In addition, the effects of grouping were consistent when the subgroups were positioned either vertically or horizontally in relation to each other. Lastly, we found that the effects of visual grouping are consistent across both opinion-and behavior/fact-based questions, although the effects appear to be greater on opinion-based questions. Our findings contribute to the increasing evidence that both verbal and visual languages influence how respondents process and respond to surveys. Because respondents interpret the verbal and graphical features of survey questionnaires as relevant to answering the survey, inadvertent or stylistic design changes can influence how respondents process and respond to survey questions.
We utilize and apply visual design theory to experimentally test ways to improve the likelihood t... more We utilize and apply visual design theory to experimentally test ways to improve the likelihood that web respondents report date answers in a particular format desired by the researcher, thus reducing possible deleterious effects of error messages or requests for corrections. These experiments were embedded in a series of web surveys of random samples of university students. We seek to examine the sequential and cumulative effects of visually manipulating the size and proximity of the answer spaces, the use of symbols instead of words, the verbal language of the question stem, and the graphical location of the symbolic instruction. Our results show that the successive series of visual language manipulations improve respondents' use of the desired format (two digits for the month and four digits for the year) from 45 percent to 96 percent. These results suggest that writing effective questions for web surveys may depend as much or more on the presentation of the answer categories/ spaces as the question wording itself.
In this overview, we discuss the current state of survey methodology in a form that is useful and... more In this overview, we discuss the current state of survey methodology in a form that is useful and informative to a general social science audience. The article covers existing challenges, dilemmas, and opportunities for survey researchers and social scientists. We draw on the most current research to articulate our points; however, we also speculate on both technological and cultural changes that currently influence or may soon affect the efficacy of different methodological choices.
Researchers who are interested in small towns and rural communities in the United States often fi... more Researchers who are interested in small towns and rural communities in the United States often find that they need to conduct their own sample surveys because many large national surveys, such as the American Community Survey, do not collect enough representative responses to make precise estimates. In collecting their own survey data, researchers face a number of challenges, such as sampling and coverage limitations. This article summarizes those challenges and tests mail and Internet methodologies for collecting data in small towns and rural communities using the U.S. Postal Service's Delivery Sequence File as a sample frame. Findings indicate that the Delivery Sequence File can be used to sample households in rural locations by sending them invitations via postal mail to respond to either paper-and-pencil or Internet surveys. Although the mail methodology is quite successful, the results for the Internet suggest that Web surveys alone exclude potentially important segments of the population of small towns and rural communities. However, Web surveys supplemented with postal questionnaires produce results quite similar to those of mail-only surveys, representing a possible cost savings for researchers who have access to Web survey capabilities.
The expertise of behavioral survey methodologists is needed in national statistical agencies beca... more The expertise of behavioral survey methodologists is needed in national statistical agencies because of the necessity of using theory and research from the social sciences to reduce survey error. In this paper various social science based explanations for measurement error and nonresponse error are described in order to illustrate the conceptual foundations of such error reduction efforts. Three roles for behavioral survey methodologists in national statistical agencies are then discussed. They include: 1) bring an error reduction perspective to bear in an influential way for all aspects of designing and implementing agency surveys; 2) bring theoretical efficiency and effectiveness to experimental tests of alternative questionnaire designs and implementation procedures through the use of theories, concepts and pretests and findings of past behavioral science research; and 3) contribute to the expanding science of survey methodology.
Previous literature has shown the effects of question wording or visual design on the data provid... more Previous literature has shown the effects of question wording or visual design on the data provided by respondents. However, few articles have been published that link the effects of question wording and visual design to the development of questionnaire design guidelines. This article proposes specific guidelines for the design of establishment surveys within statistical agencies based on theories regarding communication and visual perception, experimental research on question wording and visual design, and findings from cognitive interviews with establishment survey respondents. The guidelines are applicable to both paper and electronic instruments, and cover such topics as the phrasing of questions, the use of space, the placement and wording of instructions, the design of answer spaces, and matrices.
This paper reports results from a series of experimental manipulations of check-all-thatapply que... more This paper reports results from a series of experimental manipulations of check-all-thatapply questions in an Internet survey. One purpose of the experiments was to determine whether reversing the order of the presentation of response options resulted in order effects, including primacy and anchoring A second purpose was to determine the effects of converting check-allthat-apply questions to a forced choice-format (e.g. Yes/No for each item). We found that the existence of order effects in check-all-that-apply questions appears to depend on whether questions require respondents to use temporarily or chronically accessible information, as has been previously reported for items requiring selection of only one response option. The conversion to a forced/choice format increased the percentage of respondents answering affirmatively to each response option. Additionally, forced-choice formatted questions were unaffected by the use of more active answer categories (e.g. fan/not a fan) as opposed to the common yes/no format. Finally, the results from this survey of a random sample of 1503 University students, were quite similar to results from a previous mail survey experiment, suggesting that the response patterns observed in the current experiment result from selfadministration in general, and not from a unique characteristic of web surveys. Respondents to self-administered surveys are often asked to answer a question by checking only those response options that apply to them. For example, they might be instructed, "Please indicate which of the following sources of information you have used to find employment in the last month by checking all that apply." Each information source in the list would then be marked only if it had been used.
Recent experimental research has shown that respondents to forced-choice questions endorse signif... more Recent experimental research has shown that respondents to forced-choice questions endorse significantly more options than respondents to check-all questions. This research has challenged the common assumption that these two question formats can be used interchangeably but has been limited to comparisons within a single survey mode. In this paper we use data from a 2004 random sample survey of university students to compare the forced-choice and check-all question formats across web self-administered and telephone interviewer-administered surveys as they are commonly used in survey practice. We find that the within-mode question format effects revealed by previous research and reaffirmed in the current study appear to persist across modes as well; the telephone forced-choice format produces higher endorsement than the web check-all format. These results provide further support for the argument that the check-all and forced-choice question formats do not produce comparable results and are not interchangeable formats. Additional comparisons show that the forced-choice format performs similarly across telephone and web modes.
How to cite Complete issue More information about this article Journal's homepage in redalyc.org ... more How to cite Complete issue More information about this article Journal's homepage in redalyc.org Scientific Information System Network of Scientific Journals from Latin America, the Caribbean, Spain and Portugal Non-profit academic project, developed under the open access initiative
Antonia Aelterman Se-yuen Mak John Ainley Carmel McNaught Sonia Ben Jafaar Judith Meece Anne Boom... more Antonia Aelterman Se-yuen Mak John Ainley Carmel McNaught Sonia Ben Jafaar Judith Meece Anne Boomsma Martina R. M. Meelissen Roel Bosker R. Daniel Muijs Adriana G. Bus Rebecca Norman Derek S. P. Cheung Marie-Christine Opdenakker Geraldine Clarebout Constantinos Papanastasiou Casey Cobb Thea T. D. Peetsma Peter Daly Hans Pelgrum Jeffrey Day Ellen Reames Cees M. de Glopper Jill Salisbury-Glennon Elisabeth De Groot Shifra Schonmann Bernadet de Jager David M. Shannon Perry den Brok Charles Teddlie Don Dillman Kirsi Tirri Harrie M. C. Eijkelhof Joke Torbeyns Barry J. Fraser Kwok Tung Tsui Peter A. Frensch Jan Van Damme Femke Geijsel Eva Van de gaer John Gray Greetje P. C. van der Werf Kathy Green Karen Van Petegem Henk Guldemond Peter Van Petegem Jan-Eric Gustafsson Marcel Veenman Philip Hallinger Jean Pierre H. C. Verhaeghe Egbert G. Harskamp Ludo Verhoeven Michelle Helms Peter P. J. L. Verkoeijen Lynda Idleman Nico Verloop David Kember J. D. H. M. Vermunt Paul A. Kirschner Adrie J. Visscher Meta L. Krüger Helen M. G. Watt Leonidas Kyriakides Felicity Wikeley Yuen-kuang Cliff Liao Theo Wubbels Hans Luyten Educational Research and Evaluation Vol. 12, No. 1, February 2006, pp. 95
Survey methodology now faces an unprecedented challenge for how to collect information from sampl... more Survey methodology now faces an unprecedented challenge for how to collect information from samples of people that will provide scientifically defensible estimates of the characteristics of the population they represent. Many decades of research have shown that in order for such estimates to be made with known precision four major sources of error-coverage, sampling, measurement and nonresponse-must be controlled (Groves, 1989). Subsequent research has produced a great amount of knowledge on how those sample estimates are affected by different survey modes, sample sources, sample sizes, the failure of certain types of people to respond to survey requests, and how questions are structured and worded.Today's challenge stems from many considerations. Response rates for some survey modes, especially voice telephone in national surveys, have fallen precipitously and are not expected to recover (Dillman, Forthcoming). In addition, RDD landline surveys miss nearly half of all household...
These are challenging times for survey methodologists. The telephone, which once seemed the heir ... more These are challenging times for survey methodologists. The telephone, which once seemed the heir apparent to face-to-face interviews, is no longer an obvious choice for conducting many surveys. It is being adversely affected by the change in connectedness of telephone instruments from households to individuals and a cultural shift whereby people are able to control the telephone rather than vice versa, as was the case during its rapid rise to prominence as a mode of data collection in the late 1970s to early 1980s (Dillman 1999). The internet has not served as an immediate source of salvation. It is limited by restrictions on access, an inability to develop sample frames, and response rates that are mostly lower than those achievable by telephone (Shonlau, Fricker, and Elliott 2002). Other data-collection methodologies, from Interactive Voice Response (IVR) to classroom and home administration on computers, and mixed-mode designs are being proposed as unique solutions for specific survey topics and populations. Random sampling from defined populations is being forgone by some, replaced by modeling of data collected from volunteer panels. The current situation, seen from a methodological perspective, has elements of being both the best and worst of times. Less than a decade into the era of electronic surveying is not a time for either despair or jubilation, but I think it is an appropriate time to take a look at where survey methodology has been and where it might be going. Specifically, I will talk about changes in our uses of survey modes, new areas of needed research that result from those changes, and some implications for AAPOR. The context for these observations is set by perspectives gained over the course of my career with a variety of survey modes. These experiences don a. dillman is the Thomas S.
ñolas-como Cataluña y Euskadi-el sistema comunicativo, y también el sistema político, es muy dife... more ñolas-como Cataluña y Euskadi-el sistema comunicativo, y también el sistema político, es muy diferente a los sistemas que se han configurado en la capital del Estado, que, por cierto, tiene unas características muy singulares.
Journal of Continuing Education in the Health Professions, 2010
... Login to save citations to My List. Citation. Database: PsycINFO. [Book; Authored Book]. Inte... more ... Login to save citations to My List. Citation. Database: PsycINFO. [Book; Authored Book]. Internet, mail, and mixed-mode surveys: The tailored design method. (3rd ed). Dillman, Don A.; Smyth, Jolene D.; Christian, Leah Melani. Hoboken, NJ, US: John Wiley & Sons Inc. (2009). ...
Uploads
Papers by Don Dillman