糖心TV

An IDEA of how to get the best out of experts

This talk presents a fairly novel structured expert judgement (SEJ) protocol for quantifying parameter uncertainty using multiple experts. Generally, multiple expert opinions need to be aggregated. The two main flavours of aggregation, behavioural and mathematical, define two main classes of SEJ protocols. A third one, the so-called “mixed” approach combines aspects of both behavioural and mathematical methods. The IDEA protocol proposed here mixes specific elements from the three SEJ classes of approaches mentioned above, such that their disadvantages are minimised and their respective advantages cumulate. The acronym IDEA arises from the combination of the key features of the protocol that distinguish it from other structured elicitation procedures: it encourages experts to Investigate, Discuss, and Estimate, following which judgements are combined using mathematical Aggregation.

The experts give their individual opinions in subsequent rounds of elicitations, in a remote manner. In the first round, the experts are required to answer the questions without engaging in any (virtual) discussion with the other experts. They are then given the opportunity to discuss differences of opinion and reconcile the meaning of questions and context. The debate is remote (using an online platform) rather than face-to-face. This has the advantage of promoting the wisdom of crowds, whilst avoiding the tensions associated with group discussion between dominating personalities. The second estimate is again individual and strictly anonymous. At the end of the second round the output is a set of estimates that should further be mathematically aggregated. Several aggregation rules, many of them performance based, can be used and compared. Experts’ performance may be measured in terms of accuracy, calibration and informativness.