Calls to list polar bears as a threatened species under the United States Endangered Species Act ... more Calls to list polar bears as a threatened species under the United States Endangered Species Act are based on forecasts of substantial long-term declines in their population. Nine government reports were written to help US Fish and Wildlife Service managers decide whether or not to list polar bears as a threatened species. We assessed these reports based on evidence-based (scientific) forecasting principles. None of the reports referred to sources of scientific forecasting methodology. Of the nine, Amstrup et al.
Prior research found that people's assessments of relative competence predicted the outcome of Se... more Prior research found that people's assessments of relative competence predicted the outcome of Senate and Congressional races. We hypothesized that snap judgments of "facial competence" would provide useful forecasts of the popular vote in presidential primaries before the candidates become well known to the voters. We obtained facial competence ratings of 11 potential candidates for the Democratic Party nomination and of 13 for the Republican Party nomination for the 2008 U.S. Presidential election. To ensure that raters did not recognize the candidates, we relied heavily on young subjects from Australia and New Zealand. We obtained between 139 and 348 usable ratings per candidate between May and August 2007. The top-rated candidates were Clinton and Obama for the Democrats and McCain, Hunter, and Hagel for the Republicans; Giuliani was 9th and Thompson was 10th. At the time, the leading candidates in the Democratic polls were Clinton at 38% and Obama at 20%, while Giuliani was first among the Republicans at 28% followed by Thompson at 22%. McCain trailed at 15%. Voters had already linked Hillary Clinton's competent appearance with her name, so her high standing in the polls met our expectations. As voters learned the appearance of the other candidates, poll rankings moved towards facial competence rankings. At the time that Obama clinched the nomination, Clinton was ahead in the popular vote in the primaries and McCain had secured the Republican nomination with a popular vote that was twice that of Romney, the next highest vote-getter.
When the beach patrol raises the alarm that a shark has been sighted we know what to do, but how ... more When the beach patrol raises the alarm that a shark has been sighted we know what to do, but how should we respond to an alarm that is based on predictions of what will happen 100 years from now and the person raising the alarm tells us we must make great sacrifices now to avoid the predicted catastrophe? To answer this question, we forecast effects and outcomes of the current global warming alarm using a structured analysis of analogous situations. To do this, we searched the literature and asked experts to identify phenomena that were similar to the alarm currently being raised over dangerous manmade global warming. We obtained 71 proposed analogies. Of these, 26 met our criteria that the alarm be: (1) based on forecasts of human catastrophe arising from effects of human activity on the physical environment, (2) endorsed by experts, politicians and the media, and (3) that were accompanied by calls for strong action. None of the 26 alarms were based on scientific forecasting procedures. None of the alarming forecasts were accurate. Governments took action in 23 of the analogous situations and those actions proved to be harmful in 20. The government programs remained in place after the predicted disasters failed to materialize. The global warming alarm movement appears to be the latest manifestation of a common social phenomenon: false alarms based on unscientific forecasts of human-caused environmental disasters. We predict that the alarm over forecasts of dangerous manmade global warming will, like previous similar alarms, result in harm.
There has been surprisingly little research on how best to predict decisions in conflicts. Manage... more There has been surprisingly little research on how best to predict decisions in conflicts. Managers commonly use their unaided judgement for the task. Game theory and a disciplined use of analogies have been recommended. When tested, experts using their unaided judgement and game theorists performed no better than chance. Experts using structured analogies performed better than chance, but the most accurate forecasts were provided by simulated interaction using student role players. Twenty-one game theorists made 98 forecasts for eight diverse conflicts. Forty-one experts in conflicts made 60 solo forecasts using structured analogies and 96 solo forecasts using unaided judgement (a further seven provided collaborative forecasts only) while 492 participants made 105 forecasts in simulated interactions. Overall, one-in-three forecasts by game theorists and by experts who did not use a formal method were correct. Forecasters who used structured analogies were correct for 45 percent and forecasts from simulated interactions were correct for 62 percent of forecasts. Analysis using alternative measures of accuracy does not affect the findings. Neither expertise nor collaboration appear to affect accuracy. The findings are at odds with the opinions of experts, who expected experts to be more accurate than students regardless of the method used.
Prior research found that people's assessments of relative competence predicted the outcome of Se... more Prior research found that people's assessments of relative competence predicted the outcome of Senate and Congressional races. We hypothesized that snap judgments of "facial competence" would provide useful forecasts of the popular vote in presidential primaries before the candidates become well known to the voters. We obtained facial competence ratings of 11 potential candidates for the Democratic Party nomination and of 13 for the Republican Party nomination for the 2008 U.S. Presidential election. To ensure that raters did not recognize the candidates, we relied heavily on young subjects from Australia and New Zealand. We obtained between 139 and 348 usable ratings per candidate between May and August 2007. The top-rated candidates were Clinton and Obama for the Democrats and McCain, Hunter, and Hagel for the Republicans; Giuliani was 9 th and Thompson was 10 th. At the time, the leading candidates in the Democratic polls were Clinton at 38% and Obama at 20%, while Giuliani was first among the Republicans at 28% followed by Thompson at 22%. McCain trailed at 15%. Voters had already linked Hillary Clinton's competent appearance with her name, so her high standing in the polls met our expectations. As voters learned the appearance of the other candidates, poll rankings moved towards facial competence rankings. At the time that Obama clinched the nomination, Clinton was ahead in the popular vote in the primaries and McCain had secured the Republican nomination with a popular vote that was twice that of Romney, the next highest vote-getter.
This article introduces this JBR Special Issue on simple versus complex methods in forecasting. S... more This article introduces this JBR Special Issue on simple versus complex methods in forecasting. Simplicity in forecasting requires that (1) method, (2) representation of cumulative knowledge, (3) relationships in models, and (4) relationships among models, forecasts, and decisions are all sufficiently uncomplicated as to be easily understood by decision-makers. Our review of studies comparing simple and complex methodsincluding those in this special issue-found 97 comparisons in 32 papers. None of the papers provide a balance of evidence that complexity improves forecast accuracy. Complexity increases forecast error by 27 percent on average in the 25 papers with quantitative comparisons. The finding is consistent with prior research to identify valid forecasting methods: all 22 previously identified evidence-based forecasting procedures are simple. Nevertheless, complexity remains popular among researchers, forecasters, and clients. Some evidence suggests that the popularity of complexity may be due to incentives: (1) researchers are rewarded for publishing in highly ranked journals, which favor complexity; (2) forecasters can use complex methods to provide forecasts that support decision-makers' plans; and (3) forecasters' clients may be reassured by the incomprehensability. Clients who prefer accuracy should accept forecasts only from simple evidence-based procedures. They can rate the simplicity of forecasters' procedures using the questionnaire at simple-forecasting.com.
In important conflicts such as wars and labor-management disputes, people typically rely on the j... more In important conflicts such as wars and labor-management disputes, people typically rely on the judgment of experts to predict the decisions that will be made. We compared the accuracy of 106 forecasts by experts and 169 forecasts by novices about eight real conflicts. The forecasts of experts who used their unaided judgment were little better than those of novices. Moreover, neither group's forecasts were much more accurate than simply guessing. The forecasts of experienced experts were no more accurate than the forecasts of those with less experience. The experts were nevertheless confident in the accuracy of their forecasts. Speculating that consideration of the relative frequency of decisions across similar conflicts might improve accuracy, we obtained 89 sets of frequencies from novices instructed to assume there were 100 similar situations. Forecasts based on the frequencies were no more accurate than 96 forecasts from novices asked to pick the single most likely decision. We conclude that expert judgment should not be used for predicting decisions that people will make in conflicts. When decision makers ask experts for their opinions, they are likely to overlook other, more useful, approaches.
This article appeared in a journal published by Elsevier. The attached copy is furnished to the a... more This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier's archiving and manuscript policies are encouraged to visit: http://www.elsevier.com/copyright
We looked at evidence from comparative empirical studies to identify methods that can be useful f... more We looked at evidence from comparative empirical studies to identify methods that can be useful for predicting demand in various situations and to warn against methods that should not be used. In general, use structured methods and avoid intuition, unstructured meetings, focus groups, and data mining. In situations where there are sufficient data, use quantitative methods including extrapolation, quantitative analogies, rule-based forecasting, and causal methods. Otherwise, use methods that structure judgement including surveys of intentions and expectations, judgmental bootstrapping, structured analogies, and simulated interaction. Managers' domain knowledge should be incorporated into statistical forecasts. Methods for combining forecasts, including Delphi and prediction markets, improve accuracy. We provide guidelines for the effective use of forecasts, including such procedures as scenarios. Few organizations use many of the methods described in this paper. Thus, there are opportunities to improve efficiency by adopting these forecasting practices.
When people in conflicts can accurately forecast how others will respond, they should be able to ... more When people in conflicts can accurately forecast how others will respond, they should be able to make better decisions. Contrary to expectations, earlier research found game theorists' forecasts were less accurate than forecasts from student role players. To assess whether game theorists had been disadvantaged by the selection of conflicts, I obtained forecasts for three new conflicts of types preferred by game theory experts. As before, role-players in simulated interactions were students, and other students forecast using their judgement. Game theorists did better than previously. However, when the three new and five earlier conflicts are combined, 101 forecasts by 23 game theorists were no more accurate (31%) than 354 forecasts by students who used unaided judgement (31%). Experienced game theorists were not more accurate. Neither were those who spent more time on the task. Of 105 simulated-interaction forecasts, 62% were accurate: an average error reduction of 47% over game-theorist forecasts and a halving of error relative to the current method. Forecasts can sometimes have value without being strictly accurate. Assessing the usefulness of forecasts led to the same conclusions about the relative merits of the methods. Finally, by combining simulated interaction forecasts, accurate forecasts were obtained for seven of the eight situations.
Can game theory aid in forecasting the decision making of parties in a conflict? A review of the ... more Can game theory aid in forecasting the decision making of parties in a conflict? A review of the literature revealed diverse opinions but no empirical evidence on this question. When put to the test, game theorists' predictions were more accurate than those from unaided judgement but not as accurate as role-play forecasts. Twenty-one game theorists made 99 forecasts of decisions for six conflict situations. The same situations were described to 290 research participants, who made 207 forecasts using unaided judgement, and to 933 participants, who made 158 forecasts in active role-playing. Averaged across the six situations, 37 percent of the game theorists' forecasts, 28 percent of the unaided-judgement forecasts, and 64 percent of the role-play forecasts were correct.
Calls to list polar bears as a threatened species under the United States Endangered Species Act ... more Calls to list polar bears as a threatened species under the United States Endangered Species Act are based on forecasts of substantial long-term declines in their population. Nine government reports were written to help US Fish and Wildlife Service managers decide whether or not to list polar bears as a threatened species. We assessed these reports based on evidence-based (scientific) forecasting principles. None of the reports referred to sources of scientific forecasting methodology. Of the nine, Amstrup et al.
Prior research found that people's assessments of relative competence predicted the outcome of Se... more Prior research found that people's assessments of relative competence predicted the outcome of Senate and Congressional races. We hypothesized that snap judgments of "facial competence" would provide useful forecasts of the popular vote in presidential primaries before the candidates become well known to the voters. We obtained facial competence ratings of 11 potential candidates for the Democratic Party nomination and of 13 for the Republican Party nomination for the 2008 U.S. Presidential election. To ensure that raters did not recognize the candidates, we relied heavily on young subjects from Australia and New Zealand. We obtained between 139 and 348 usable ratings per candidate between May and August 2007. The top-rated candidates were Clinton and Obama for the Democrats and McCain, Hunter, and Hagel for the Republicans; Giuliani was 9th and Thompson was 10th. At the time, the leading candidates in the Democratic polls were Clinton at 38% and Obama at 20%, while Giuliani was first among the Republicans at 28% followed by Thompson at 22%. McCain trailed at 15%. Voters had already linked Hillary Clinton's competent appearance with her name, so her high standing in the polls met our expectations. As voters learned the appearance of the other candidates, poll rankings moved towards facial competence rankings. At the time that Obama clinched the nomination, Clinton was ahead in the popular vote in the primaries and McCain had secured the Republican nomination with a popular vote that was twice that of Romney, the next highest vote-getter.
When the beach patrol raises the alarm that a shark has been sighted we know what to do, but how ... more When the beach patrol raises the alarm that a shark has been sighted we know what to do, but how should we respond to an alarm that is based on predictions of what will happen 100 years from now and the person raising the alarm tells us we must make great sacrifices now to avoid the predicted catastrophe? To answer this question, we forecast effects and outcomes of the current global warming alarm using a structured analysis of analogous situations. To do this, we searched the literature and asked experts to identify phenomena that were similar to the alarm currently being raised over dangerous manmade global warming. We obtained 71 proposed analogies. Of these, 26 met our criteria that the alarm be: (1) based on forecasts of human catastrophe arising from effects of human activity on the physical environment, (2) endorsed by experts, politicians and the media, and (3) that were accompanied by calls for strong action. None of the 26 alarms were based on scientific forecasting procedures. None of the alarming forecasts were accurate. Governments took action in 23 of the analogous situations and those actions proved to be harmful in 20. The government programs remained in place after the predicted disasters failed to materialize. The global warming alarm movement appears to be the latest manifestation of a common social phenomenon: false alarms based on unscientific forecasts of human-caused environmental disasters. We predict that the alarm over forecasts of dangerous manmade global warming will, like previous similar alarms, result in harm.
There has been surprisingly little research on how best to predict decisions in conflicts. Manage... more There has been surprisingly little research on how best to predict decisions in conflicts. Managers commonly use their unaided judgement for the task. Game theory and a disciplined use of analogies have been recommended. When tested, experts using their unaided judgement and game theorists performed no better than chance. Experts using structured analogies performed better than chance, but the most accurate forecasts were provided by simulated interaction using student role players. Twenty-one game theorists made 98 forecasts for eight diverse conflicts. Forty-one experts in conflicts made 60 solo forecasts using structured analogies and 96 solo forecasts using unaided judgement (a further seven provided collaborative forecasts only) while 492 participants made 105 forecasts in simulated interactions. Overall, one-in-three forecasts by game theorists and by experts who did not use a formal method were correct. Forecasters who used structured analogies were correct for 45 percent and forecasts from simulated interactions were correct for 62 percent of forecasts. Analysis using alternative measures of accuracy does not affect the findings. Neither expertise nor collaboration appear to affect accuracy. The findings are at odds with the opinions of experts, who expected experts to be more accurate than students regardless of the method used.
Prior research found that people's assessments of relative competence predicted the outcome of Se... more Prior research found that people's assessments of relative competence predicted the outcome of Senate and Congressional races. We hypothesized that snap judgments of "facial competence" would provide useful forecasts of the popular vote in presidential primaries before the candidates become well known to the voters. We obtained facial competence ratings of 11 potential candidates for the Democratic Party nomination and of 13 for the Republican Party nomination for the 2008 U.S. Presidential election. To ensure that raters did not recognize the candidates, we relied heavily on young subjects from Australia and New Zealand. We obtained between 139 and 348 usable ratings per candidate between May and August 2007. The top-rated candidates were Clinton and Obama for the Democrats and McCain, Hunter, and Hagel for the Republicans; Giuliani was 9 th and Thompson was 10 th. At the time, the leading candidates in the Democratic polls were Clinton at 38% and Obama at 20%, while Giuliani was first among the Republicans at 28% followed by Thompson at 22%. McCain trailed at 15%. Voters had already linked Hillary Clinton's competent appearance with her name, so her high standing in the polls met our expectations. As voters learned the appearance of the other candidates, poll rankings moved towards facial competence rankings. At the time that Obama clinched the nomination, Clinton was ahead in the popular vote in the primaries and McCain had secured the Republican nomination with a popular vote that was twice that of Romney, the next highest vote-getter.
This article introduces this JBR Special Issue on simple versus complex methods in forecasting. S... more This article introduces this JBR Special Issue on simple versus complex methods in forecasting. Simplicity in forecasting requires that (1) method, (2) representation of cumulative knowledge, (3) relationships in models, and (4) relationships among models, forecasts, and decisions are all sufficiently uncomplicated as to be easily understood by decision-makers. Our review of studies comparing simple and complex methodsincluding those in this special issue-found 97 comparisons in 32 papers. None of the papers provide a balance of evidence that complexity improves forecast accuracy. Complexity increases forecast error by 27 percent on average in the 25 papers with quantitative comparisons. The finding is consistent with prior research to identify valid forecasting methods: all 22 previously identified evidence-based forecasting procedures are simple. Nevertheless, complexity remains popular among researchers, forecasters, and clients. Some evidence suggests that the popularity of complexity may be due to incentives: (1) researchers are rewarded for publishing in highly ranked journals, which favor complexity; (2) forecasters can use complex methods to provide forecasts that support decision-makers' plans; and (3) forecasters' clients may be reassured by the incomprehensability. Clients who prefer accuracy should accept forecasts only from simple evidence-based procedures. They can rate the simplicity of forecasters' procedures using the questionnaire at simple-forecasting.com.
In important conflicts such as wars and labor-management disputes, people typically rely on the j... more In important conflicts such as wars and labor-management disputes, people typically rely on the judgment of experts to predict the decisions that will be made. We compared the accuracy of 106 forecasts by experts and 169 forecasts by novices about eight real conflicts. The forecasts of experts who used their unaided judgment were little better than those of novices. Moreover, neither group's forecasts were much more accurate than simply guessing. The forecasts of experienced experts were no more accurate than the forecasts of those with less experience. The experts were nevertheless confident in the accuracy of their forecasts. Speculating that consideration of the relative frequency of decisions across similar conflicts might improve accuracy, we obtained 89 sets of frequencies from novices instructed to assume there were 100 similar situations. Forecasts based on the frequencies were no more accurate than 96 forecasts from novices asked to pick the single most likely decision. We conclude that expert judgment should not be used for predicting decisions that people will make in conflicts. When decision makers ask experts for their opinions, they are likely to overlook other, more useful, approaches.
This article appeared in a journal published by Elsevier. The attached copy is furnished to the a... more This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier's archiving and manuscript policies are encouraged to visit: http://www.elsevier.com/copyright
We looked at evidence from comparative empirical studies to identify methods that can be useful f... more We looked at evidence from comparative empirical studies to identify methods that can be useful for predicting demand in various situations and to warn against methods that should not be used. In general, use structured methods and avoid intuition, unstructured meetings, focus groups, and data mining. In situations where there are sufficient data, use quantitative methods including extrapolation, quantitative analogies, rule-based forecasting, and causal methods. Otherwise, use methods that structure judgement including surveys of intentions and expectations, judgmental bootstrapping, structured analogies, and simulated interaction. Managers' domain knowledge should be incorporated into statistical forecasts. Methods for combining forecasts, including Delphi and prediction markets, improve accuracy. We provide guidelines for the effective use of forecasts, including such procedures as scenarios. Few organizations use many of the methods described in this paper. Thus, there are opportunities to improve efficiency by adopting these forecasting practices.
When people in conflicts can accurately forecast how others will respond, they should be able to ... more When people in conflicts can accurately forecast how others will respond, they should be able to make better decisions. Contrary to expectations, earlier research found game theorists' forecasts were less accurate than forecasts from student role players. To assess whether game theorists had been disadvantaged by the selection of conflicts, I obtained forecasts for three new conflicts of types preferred by game theory experts. As before, role-players in simulated interactions were students, and other students forecast using their judgement. Game theorists did better than previously. However, when the three new and five earlier conflicts are combined, 101 forecasts by 23 game theorists were no more accurate (31%) than 354 forecasts by students who used unaided judgement (31%). Experienced game theorists were not more accurate. Neither were those who spent more time on the task. Of 105 simulated-interaction forecasts, 62% were accurate: an average error reduction of 47% over game-theorist forecasts and a halving of error relative to the current method. Forecasts can sometimes have value without being strictly accurate. Assessing the usefulness of forecasts led to the same conclusions about the relative merits of the methods. Finally, by combining simulated interaction forecasts, accurate forecasts were obtained for seven of the eight situations.
Can game theory aid in forecasting the decision making of parties in a conflict? A review of the ... more Can game theory aid in forecasting the decision making of parties in a conflict? A review of the literature revealed diverse opinions but no empirical evidence on this question. When put to the test, game theorists' predictions were more accurate than those from unaided judgement but not as accurate as role-play forecasts. Twenty-one game theorists made 99 forecasts of decisions for six conflict situations. The same situations were described to 290 research participants, who made 207 forecasts using unaided judgement, and to 933 participants, who made 158 forecasts in active role-playing. Averaged across the six situations, 37 percent of the game theorists' forecasts, 28 percent of the unaided-judgement forecasts, and 64 percent of the role-play forecasts were correct.
Uploads
Papers by Kesten Green