How Do You Identify Skill?

Many people involved in financial markets are engaged in a perpetual quest to identify skill – that is the attempt to seek out individuals or teams with the expertise to deliver abnormally strong investment returns. Whilst this is an understandable endeavour, it is also incredibly difficult.  In many domains and activities we can simply use outcomes as an effective proxy for skill, but in environments where uncertainty and randomness exert a significant influence, results alone can be woefully misleading.  Whilst an Elo rating may give you a robust guide to an individual’s ability to play chess; a track record of outperformance for an active manager will offer limited guidance on the underlying skill exhibited.

The vast, adaptive and reflexive nature of financial markets means that even if skill exists in certain areas, recognising it is hugely problematic. The complexity of the task means that investors typically default to a simple outcome driven approach – an effective heuristic in many other areas.

Given that focusing on outcomes alone is inadequate in an investment context, how should we approach locating skill in an activity where randomness heavily skews the results?   Rather than focus on one element, I think there are six important inter-related components that need to be considered: Specification, Calibration, Intention, Path, Outcome and Replication.

I will cover each in turn utilising a golfing analogy – although I don’t play the game, it is an activity that does incorporate both luck (less) and skill (more), and will hopefully serve to simplify the idea.

There are two golfers (Golfer A and Golfer B) both have taken one shot at par 3 hole and landed the ball very close to the flag – let’s say 3 feet. How do we determine whether each player has golfing skill?

Outcomes: If we were simply judging outcomes alone we might say that both golfers possess skill, as they have both produced excellent results.

Path: Understanding the path (how a result was achieved) can give us far richer information. Golfer A’s shot went arrow straight at the flag, Golfer B sliced their shot and it rebounded off a tree and onto the green.  Given this new information, we are emboldened in our view that Golfer A has skill, but now we are doubtful that Golfer B does – it looks as if they have just enjoyed a significant amount of luck.

Intention: It is very dangerous to assume that an individual has skill simply from observing an activity – if you don’t understand what they were trying to achieve beforehand.  If we know that Golfer A was attempting to hit his shot at the flag near which they landed their ball then we can have increased confidence that they possess skill.  But what if we knew that Golfer B was actually attempting to hit their ball onto the green after ricocheting off a tree?  Rather than believe that they had just been lucky, we might consider that they have superior skill to Golfer A because they performed a more difficult task.

Replication: Samples of one are never a good guide to skill and the more randomness in an activity the more evidence you require.  Although we might have a strong inclination that Golfer A and B both possess skill – with one example each we are incredibly vulnerable to being fooled by random occurrences.

Specification: When seeking to define skill, we need to be very specific about the activity in which someone possesses it.  Even if we witness both Golfer A and Golfer B repeat the exact same feat on numerous occasions – we can only be confident that they have skill in that precise task – we may infer that they are skilled golfers, but they might be terrible at putting, for example, a particular aspect of the game on which we have no evidence.

Calibration: All activities sit somewhere on the luck and skill continuum, and it is important to have a perspective on how much randomness and complexity there is in an activity before making any judgments about skill.  For example, landing a plane is dominated by skill with a slither of luck involved – if I witness an individual landing a plane successfully it gives me far greater confidence that they have skill in that task, than the confidence I might gain from watching someone hit a single good golf shot.  Trying to correctly calibrate the randomness inherent in an activity helps you to understand how much value there might be in the outcomes alone.

When chance is involved in an activity then we need to rely less on results. As we can see from the golfing example, understanding the different elements of the process can transform our view on whether we are observing skill or randomness.  When we are working to identify skill we should always be able to answer, at least, the following questions:

Specification: What is the precise activity in which we are attempting to identify skill?

Calibration: How much luck or randomness do we think is involved in the activity?

Intention: What is the objective of the activity?

Path: How has the objective been reached?

Outcome: What was the overall result?

Replication: How often has this process led to the same outcome?

In the investment industry we give pre-eminence to outcomes when determining skill. Even when we incorporate other factors, our perspective is often biased by the strong priors we develop after initially observing performance – if we see strong performance; we assume skill must be involved.  We are also prone to assume that apparent skill in one specific aspect translates across the entire spectrum of investment activities – someone is often considered a ‘good / great investor’- good at what, exactly?

Although skewed incentives and our obsession with outcomes make it incredibly demanding, the only way to even attempt to successfully identify skill is to understand not what the outcomes were, but precisely how they have been achieved.