A novel approach to reducing uncertainty: The group Delphi

2000

The Delphi technique provides different opportunities to researchers than survey research. Essential components of the Delphi technique include the communication process, a group of experts, and essential feedback. This paper provides the foundations of the Delphi Technique, discusses its strengths and weaknesses, explains the use and stages followed, discusses panel selection, and explains how consensus among participants is reached.

Risk Analysis, 2002

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quanti®cation of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quanti®cation task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary ®eld of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary ®elds of interest.

2012

Many environmental and risk management decisions are made jointly by technical experts and members of the public. Frequently, their task is to select from among management alternatives whose outcomes are subject to varying degrees of uncertainty. Although it is recognized that how this uncertainty is interpreted can significantly affect decision-making processes and choices, little research has examined similarities and differences between expert and public understandings of uncertainty.

Environmental Modelling & Software, 2012

A decision-making process focusing on environmental issues is extremely complex because of the intricacy of the real-world systems. Such systems are subjected to many uncertain events, which make planning, modeling, and predicting performances and treatment inherently complicated. Typically, a decision-making process focusing on environmental problems is ill structured, uncertain, vague, and multidimensional and is often based on the opinions of experts with different viewpoints. A common problem is how to aggregate the opinions of experts, which might be diverse and sometimes even opposing. This paper presents a new method for aggregating experts' opinions and introduces a new aggregation operator MaxAgM, based on Shannon entropy, which maximizes the agreement of experts' opinions. Our method can be applied toward aggregating expert proposals that were expressed by crisp as well as fuzzy quantities to propose a binary solution or to estimate a numerical value of some parameter. A specialized software package MaxAgr was developed to optimize agreement drawn from experts' proposals. Application of the method and the software is illustrated in a case study on flood risk management.

Environmental Modelling & Software, 2012

The inevitable though frequently informal use of expert opinion in modelling, the increasing number of models that incorporate formally expert opinion from a diverse range of experience and stakeholders, arguments for participatory modelling and analytic-deliberative-adaptive approaches to managing complex environmental problems, and an expanding but uneven literature prompt this critical review and analysis. Aims are to propose common definitions, identify and categorise existing concepts and practice, and provide a frame of reference and guidance for future environmental modelling. The extensive literature review and classification conducted demonstrate that a broad and inclusive definition of experts and expert opinion is both required and part of current practice. Thus an expert can be anyone with relevant and extensive or in-depth experience in relation to a topic of interest. The literature review also exposes informal model assumptions and modeller subjectivity, examines in detail the formal uses of expert opinion and expert systems, and critically analyses the main concepts of, and issues arising in, expert elicitation and the modelling of associated uncertainty. It is noted that model scrutiny and use of expert opinion in modelling will benefit from formal, systematic and transparent procedures that include as wide a range of stakeholders as possible. Enhanced awareness and utilisation of expert opinion is required for modelling that meets the informational needs of deliberative fora. These conclusions in no way diminish the importance of conventional science and scientific opinion but recognise the need for a paradigmatic shift from traditional ideals of unbiased and impartial experts towards unbiased processes of expert contestation and a plurality of expertise and eventually models. Priority must be given to the quality of the enquiry for those responsible for environmental management and policy formulation, and this review emphasises the role for science to maintain and enhance the rigour and formality of the information that informs decision making.

Technological Forecasting and Social Change, 2021

We examined whether giving feedback to participants in a Delphi study about the level of agreement across the expert panel had an effect on opinion change between rounds. We also considered the potential influence of participants' sociodemographic and professional characteristics. Five three-round Delphi studies were conducted independently, in which a total of 628 mental health experts responded to all three rounds. In each study, participants had to decide, based on their experience, whether a series of categories were relevant. The percentage of group agreement (i.e., percentage of participants who considered each category as relevant) in round 2 was shown as feedback in round 3, and responses in rounds 2 and 3 were considered to analyze opinion change. Results showed that when the feedback given in round 3 indicated that ≥75% of experts considered a category to be relevant, there was a further shift in opinion towards the group opinion (i.e., the category then yielded even greater consensus), whereas if the feedback indicated <75% group agreement, individual opinions tended to shift against the group opinion (i.e., consensus over the category decreased). Neither sociodemographic nor professional variables had a significant effect in explaining opinion shift. These results show that in Delphi studies, feedback has an influence on individual responses and the achievement of consensus.

Journal of the Royal Statistical Society: Series D (The Statistician), 2000

This paper reports an experiment in which 32 experts at three water companies quanti®ed their opinions about two problems: the cost of refurbishing speci®ed pumping stations and the length of unrecorded S24 sewers in four towns. Medians and both unconditional and conditional assessments of quantiles were elicited and assessments were compared with reality and given scores. The quantiles that experts were asked to assess varied and the experts differed in the extent of their relevant background knowledge. The in¯uences of these factors on standard deviations and scores are examined. The main focus is on modelling methods and ways of using the elicited assessments to form subjective probability distributions. We consider ®tting log-normal distributions to model asymmetric distributions and also examine models to relate a quantity's assessed standard deviation and interquantile range to its size, ®nding that interquantile ranges can be modelled more accurately. In addition, different approaches to separating opinions about components of variance are evaluated and compared.

The Delphi technique is a widely used and accepted method for gathering data from respondents within their domain of expertise. The technique is designed as a group communication process which aims to achieve a convergence of opinion on a specific real-world issue. The Delphi process has been used in various fields of study such as program planning, needs assessment, policy determination, and resource utilization to develop a full range of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by using a series of questionnaires delivered using multiple iterations to collect data from a panel of selected subjects. Subject selection, time frames for conducting and completing a study, the possibility of low response rates, and unintentionally guiding feedback from the respondent group are areas which should be considered when designing and implementing a Delphi study.

Sustainability

Many of the techniques for making decisions, including land use, depend on the weight assigned for each criterion. These criteria can be based on a panel of experts’ opinions, who assess certain decisions’ influence on the final objectives. These opinions should be contrasted to decide if they are used or select the ones used to achieve an internal coherence. In this study, we evaluate the responses provided by an expert panel in the context of future environmental management of an agroforestry territory in the Salta Province (Argentina). The experts belong to different entities in the studied area, such as Universities, Research Centers, Administrative Authorities, Associations, and non-governmental organizations. They evaluated five productive techniques’ influence on 31 criteria related to environmental, social, and economic consequences. The Kendall’s Tau correlation coefficient between each pair of experts’ opinions is proposed to measure the rate of agreement among the expert ...

Cahiers du C3ED, 2006

This article proposes a selection of tools to assess the nature and the extent of uncertainty linked to knowledge production and use, in order to integrate uncertainty in processes of environmental evaluation.

Environmental Science & Policy, 2009

Land Use Policy, 2006

The combination of deliberative and analytical methods is supposed to have a high potential for the resolution of environmental conflicts. But selecting methods and tools for a specific case often remains nebulous. In this article, we evaluate different deliberative and analytical methods according to characteristics defined by Wittmer et al. (2004). We put a special focus on multi-criteria methods as these analytical methods are especially suited for participation. This evaluation identifies strengths and weaknesses of methods, and can ground the selection of one specific method (or the combination of several) on a systematic basis. A brief definition of the methods reported in this issue is given and these are evaluated. The next step consists of identifying which characteristics are most important in a specific case. Applying these two steps can guide the selection of methods for resolving environmental conflicts. We conclude that the combination of participatory and multi-criteria methods generally provides a good starting point for conflict facilitators looking for methods supporting the decision-aid process. r

2009

The daily operation of Wastewater Treatment Plants (WWTPs) in industrialized areas is of particular concern because of the severe problems that can occur in the WWTP caused by the incoming inflow, which in turn may cause an ecological imbalance in the fluvial ecosystem. In order to minimize the environmental impact caused by the industrial wastewater discharges, guidelines and regulations exists. However, due to the complexity of the domain, there are still no golden standards by which to decide whether a WWTP can cope with wastewater discharges, and so strict adherence to regulations may not always be convenient. Special circumstances may motivate to accept discharges that are above established thresholds or to reject discharges that comply with guidelines. Nonetheless, because of the criticality of the actions to be taken, such decisions require to be well justified. Hence, in this work it is proposed the use of the argumentation-based model Pro- CLAIM to provide a more flexible decision making process, in which expertise can deliberate whether an industrial wastewater can safely be discharged into a WWTP, and thus adapt each decision to the particular circumstance. To ensure a safe decision, agents’ given arguments for or against the industrial spill are evaluated accounting for the domain guidelines and regulations, for similar past cases and for confidence in the expertise’s assessments.

2017

Delphi is a powerful technique used to seek answers to appropriate questions. The purpose of this paper is to provide an overview of the Delphi technique as a research method. This paper discusses the scientific merit of the Delphi technique by investigating on 41 studies of Journal of Agricultural Education from 1981 to 2013, and 2 studies of Journal of Agricultural Science and Technology. The results showed that there is no general agreement on using indexes in different rounds of Delphi technique; however, according to the frequencies of using indexes in different studies, the following suggestions are presented. The favourable number of panel of experts is between 10 to 20. Purposive sampling method is used for selecting the panel members. Usually a three rounds of Delphi method is used. One question is designed in round one. Mean and standard deviation indexes are used for passing from round two to round three and agreement level of 70 present is used for achieving expert&#39;s...