Imagine you are in a team meeting discussing a potential investment with three colleagues, you ask them how probable it is that your investment thesis for a particular position comes to fruition, each of them states that they see it as ‘likely’.  In an alternate universe, you are in the same situation the only difference being the responses from your colleagues – on this occasion they each say ‘60%’.  Does your colleagues’ shift from a verbal to numeric expression of probability impact your confidence in the investment decision?  A new paper from Robert Mislavsky and Celia Gaertig contends that it would – their research suggests that when we are given numeric probability forecasts we average them and when given verbal forecasts we count them.  A succession of ‘sixty percents’ leads you to a 60% average, whereas a similar number of ‘likely’ responses sees your own view become ‘very likely’.

We often talk in probabilistic terms without realising it – when we state something is ‘likely’ or ‘very likely’ we are expressing some form of view on the probability of an occurrence, although it is admittedly a vague one.  Research around this area has typically focused on the comparison and translation of verbal probability expressions into numeric ones, and vice-versa – when we say something is unlikely, what probability do we actually mean?   As Mislavsky and Gaertig note in their paper, verbal probabilities have the benefit of being clear in their direction (you can tell if it is positive or negative) but suffer from imprecision, whereas numeric probabilities are specific but the direction is not always clear (whether a 45% probability is positive depends on the context).

Mislavsky and Gaertig’s research develops the thinking around this subject by moving on from identifying specific differences between individual verbal and numeric probability expressions, and showing that there is a material change in outcome when we combine a number of verbal probabilities, compared to when we combine a selection of numeric probabilities.  Their research incorporates a range of experiments (7 in total) wherein participants were required to make a decision or predict an outcome after receiving one or two expert forecasts – these forecasts were either both numeric or both verbal.

For example, in their second study, participants were provided with some details about a company and asked to judge how likely it was that its share price would be higher in a year’s time.  Some participants received expert guidance from advisers in numeric form and some from advisers in verbal form.  The results of this study – which were consistent with all the experiments carried out in the research –  was that “participants became more likely to make extreme forecasts as they saw additional advisor forecasts in the verbal condition but less likely to do so in the numeric condition”.  We can see this in the chart below:

The predictions of the participants clearly became more extreme when they received an additional verbal forecast but not when an additional numeric forecast was provided.  By ‘extreme forecast’ the authors mean when a participant’s forecast is in excess of that given by either adviser.  Similar results were observed when the study moved from looking at a hypothetical stock price, to predictions of Major League Baseball games with probabilities given by genuine experts.

There is clear evidence from the study that the verbal probabilities lead to a counting process, whereas numeric probabilities are averaged. There are good reasons for both approaches – counting works on the basis that each adviser is providing new information, whereas averaging is prudent if we assume each forecast is founded on the same information.  There is no requirement, however, to associate the different expressions of probability with different processes for their combination – two 60% forecasts could just as easily be driven by different information as two ‘likelys’.  So what is happening?

The authors conclude the paper by reviewing and largely discounting a range of potential explanations (I would urge everyone to read the paper directly).  My best guess of the cause of the phenomenon would be a combination of some of the factors mentioned by the authors, in particular how individuals are liable to perceive numeric and verbal probabilities in different fashions.  Numeric forecasts feel precise and objective, and more consistent with an ‘outside view’ driven by the base rate or reference class – more likely to contain all relevant information.  Contrastingly, verbal probabilities seem personal and subjective, more akin to an ‘inside view’ where an individual providing a forecast will be doing so based on their own unique knowledge or perspective – therefore an additive approach can seem justified.

This idea is pure speculation about which the authors are sceptical, however, whilst the drivers of this contrasting approach to combining probabilities are uncertain; the results, from this initial study at least, are clear, and there is an important lesson for investors to heed.  It is crucial to consider not only the type of guidance and advice we are receive when informing a decision, but how it is being expressed.  This is relevant whether it relates to an individual’s decision using a range of external information sources, or a team based decision making process where we are seeking to synthesise the insights of a number of individuals into a single view.

—

Mislavsky, R., & Gaertig, C. (2019). Combining Probability Forecasts: 60% and 60% Is 60%, but Likely and Likely Is Very Likely. Available at SSRN 3454796.