Of Exams, Luck and the Paradox of Skill

exam

In last week’s column, I wrote about the role that luck, skill and hard work, play in exams. In this column, I plan to get into a little more detail on the issue.

Over the last few years, the media has made it a habit to splash the pictures of toppers of competitive exams as well as board exams (10th and 12th standards). Other than the fact that any sort of success needs to be recognised, such columns make for an inspirational read, particularly in cases where the toppers come from a poor family.

When it comes to competitive exams (from engineering exams to UPSC exams), there are magazines which interview toppers, in the hope of finding out the formula for success, so that their readers can benefit. And typically, most such news stories and interviews have more or less standard reasons being offered for success. These are hard work, family support and following a regular routine.

Of course, topping exams needs hard work and family support. But are these the only reasons? And if that is the case, how come two equally intelligent candidates, putting in the same amount of hard work and having the same level of family support, don’t perform at the same level in any exam? Because there is something known as the paradox of skill at work.

As Michael Mauboussin writes in The Success Equation—Untangling Skill and Luck in Business, Sports and Investing: “As skill improves, performance becomes more consistent, and therefore luck becomes more important… In other words, if everyone gets better at something, luck plays a more important role in determining who wins.”

Mauboussin offers the example of a company. As he writes: “A company can improve its absolute performance, for example but it will remain at a competitive parity if its rivals do the same.” In this situation whether the company does better than its rivals, depends on luck. As Mauboussin writes: “When everyone in business, sports, and investing copies the best practices of others, luck plays a greater role in how well they do.”

How does this apply in the context of exams? Most people prepare for exams these days by going to coaching institutes and if not that, at least using study material provided by coaching institutes. This is typically true more for competitive exams. But it is also true for board as well as BA/BSc/BCom exams in many states.

Given this, a significantly large pool of candidates which has access to the same study material and is also more or less equal on other parameters, faces the paradox of skill. In this situation, who comes out on top or even qualifies in a competitive exam, depends on their luck on the day of the exam.

Let me give you an example from my life. When I first wrote the Common Aptitude Test (CAT) for admission into the IIMs and other MBA colleges, I had prepared decently for the exam. The city that I grew up in did not have a CAT exam centre. So, I had to go to another city to write the exam. I spent a sleepless night in the hotel overnight. And this clearly had an impact on my performance in the exam.

If the examination centre had been in the same city that I grew up in, my performance in the exam would have been significantly better. But this was how the luck of the draw turned out.

The same logic applies to toppers as well. Of course, they need to work hard, but they also need to be lucky on day of the exam. This could mean anything from sleeping well overnight to being able to reach the exam centre on time to not becoming obsessed with a question they are not able to solve.

The media focus on the toppers does injustice to many others who do not come out on top, but are equally intelligent. It’s just that on the day of the exam things didn’t work out as well for them, as they did for the toppers. And there is no second chance.

The column originally appeared on June 28, 2017 in the Bangalore Mirror.

 

 

The curious case of Mr Jain

prashant jainVivek Kaul

 Sometime in late October I went to meet my investment advisor. During the course of our discussion he suggested that my portfolio was skewed towards HDFC Mutual Fund and it would be a good idea to move some money out of it, into other funds.
Don’t put all your eggs in one basket” is an old investment adage. While, I try to follow it, I also like to believe that if the basket is good enough, it makes sense to put more eggs in that basket than other baskets.
HDFC Mutual Fund has been one of the few consistent performers in the Indian mutual fund space. And a major reason for the same has been Prashant Jain, the chief investment officer of the fund, who has been with it for nearly two decades.
Jain has been a star performer and due to his reputation the fund has seen a huge inflow of money into its various schemes. Some of these schemes HDFC Prudence, HDFC Equity and HDFC Top 200 became very big in that process.
These schemes haven’t done very well over the last three years. Their performance has been significantly worse in comparison to other schemes in their respective categories(
Value Research has downgraded them to three star funds from being five star funds earlier). And this has surprised many people. “How can Prashant Jain not perform?” is a question close observers of the mutual fund industry in India have been asking.
One explanation that people seem to have come up with is the fact that the size of the schemes have become big, making it difficult for Jain to generate significant return. This is a theory that is globally accepted, where the size of a scheme is believed to be inversely proportional to the return it generates.
As Jason Zweig points out in the commentary to Benjamin Graham’s all time investment classic, 
The Intelligent Investor, “As a (mutual) fund grows, it fees become more lucrative – making its managers reluctant to rock the boat. The very risk that managers took to generate their initial high returns could now drive the investors away — and jeopardise all that fee income. So the biggest funds resemble a herd of identical and overfed sheep, all moving in sluggish lockstep, all saying “Baaaa” at the same time.”
While this may be a reason for the underperformance of the schemes managed by Jain, it is not easy to prove this conclusively. Jain feels there is no correlation between size and performance of a scheme, or so he told the 
Forbes India magazine in a recent interview. He pointed out that there are no large mutual fund schemes in India, and the largest scheme is less than 0.2% of the market capitalisation, which I guess is a fair point to make.
So how does one explain the fact that Prashant Jain is not doing as well as he used to in the past. John Allen Paulos possibly has an explanation for it in his book 
A Mathematician Plays the Stock Market. As he writes “A different argument points out to the near certainty of some stocks, funds, or analysts doing well over an extended period of time.”
Paulos offers an interesting thought experiment to make his point. As he writes “Of 1000 stocks (or funds or analysts), for example, roughly 500 might be expected to outperform the market next year simply by chance, say by the flipping of a coin. Of these 500, roughly 250 might be expected to do well for a second year. And of these 250, roughly 125 might be expected to continue the pattern, doing well three years in a row simply by chance. Iterating in this way, we might reasonably expect there to be a stock (or fund or analyst) among the thousand that does well for ten consecutive years by chance alone.”
But one day this winning streak comes to an end. And the same seems to have happened to Prashant Jain. In fact, William Miller who ran the Legg Mason Value Trust fund in the United States, beat the broader market every year from 1991 to 2005. In 2006, his luck finally ran out. Miller once explained his winning streak by saying “As for the so-called streak…We’ve been lucky. Well, maybe it’s not 100% luck—maybe 95% luck.”
If Miller was lucky so was Jain. Any significant deviation from the norm does not last forever. As Nassim Nicholas Taleb writes in 
Fooled by Randomness “In real life, the larger the deviation from the norm, the larger the probability of it coming from luck rather than skills…The “reversion” for the large outliers is what has been observed in history and explained as regression to the mean. Note the larger the deviation, the more important its effect.”
This is not to suggest that Jain’s performance has only been because of luck. Not at all. But it was luck that pushed him up to the top of the charts. Luck was the “icing” on the cake.
Michael Mauboussin discusses a very interesting concept called the paradox of skill in his book 
The Success Equation – Untangling Skill and Luck in Business, Sports, and Investing. “As skill improves, performance becomes more consistent, and therefore luck becomes more important,” is how Mauboussin defines the paradox of skill.
The Olympic marathon is a very good example of the same. Men run the race today about 26 minutes faster than they did 80 years back. Also, in 1932, the difference between the man who won the race and the man who came in twentieth was 40 minutes. Now its less than 10 minutes.
Now the question is h
ow does this apply to investing? “As the market is filled with participants who are smart and have access to information and computing power, the variance of skill will decline. That means that stock price changes will be random and those investors who beat the market can chalk up their success to luck. And the evidence shows that the variance in mutual fund returns has shrunk over the past 60 years, just as the paradox of skill would suggest,” says Mauboussin. “I want to be clear that I believe that differential skill in investing remains, and that I don’t believe that all results are from randomness. But there’s little doubt that markets are highly competitive and that the basic sketch of the paradox of skill applies,” he adds.
And that is what best explains the curious case of Prashant Jain and the recent non performance of the mutual fund schemes that he manages.
The column originally appeared in the Wealth Insight magazine edition of December, 2013 

(Vivek Kaul is the author of Easy Money. He tweets @kaul_vivek) 

People should listen to market experts for entertainment, not elucidation.


Michael J. Mauboussin is Chief Investment Strategist at Legg Mason Capital Management in the United States. He is also the author of bestselling books on investing like Think Twice: Harnessing the Power of Counterintuition and More Than You Know: Finding Financial Wisdom in Unconventional Places. His latest book The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing is due later this year. In this interview he speaks to Vivek Kaul on the various aspects of luck, skill and randomness and the impact they have on business, life and investing.
Excerpts:

How do you define luck?
The way I think about it, luck has three features. It happens to a person or organization; can be good or bad; and it is reasonable to believe that another outcome was possible. By this definition, if you win the lottery you are lucky, but if you are born to a wealthy family you are not lucky—because it is not reasonable to believe that any other outcome was possible—but rather fortunate.
How is randomness different from luck?
I like to distinguish, too, between randomness and luck. I like to think of randomness as something that works at a system level and luck on a lower level. So, for example, if you gather a large group of people and ask them to guess the results of five coin tosses, randomness tells you that some in the group will get them all correct. But if you get them all correct, you are lucky.
And what is skill?
For skill, the dictionary says the “ability to use one’s knowledge effectively and readily in execution or performance.” I think that’s a good definition. The key is that when there is little luck involved, skill can be honed through deliberate practice. When there’s an element of luck, skill is best considered as a process.
You have often spoken about the paradox of skill. What is that?
The paradox of skill says that as competitors in a field become more skillful, luck becomes more important in determining results. The key to this idea is what happens when skill improves in a field. There are two effects. First, the absolute level of ability rises. And second, the variance of ability declines.
Could you give us an example?
One famous example of this is batting average in the sport of baseball. Batting average is the ratio of hits to at-bats. It’s somewhat related to the same term in cricket. In 1941, a player named Ted Williams hit .406 for a season, a feat that no other player has been able to match in 70 years. The reason, it turns out, is not that no players today are as good as Williams was in his day—they are undoubtedly much better. The reason is that the variance in skill has gone down. Because the league draws from a deeper pool of talent, including great players from around the world, and because training techniques are vastly improved and more uniform, the difference between the best players and the average players within the pro ranks has narrowed. Even if you assume that luck hasn’t changed, the variance in batting averages should have come down. And that’s exactly what we see. The paradox of skill makes a very specific prediction. In realms where there is no luck, you should see absolute performance improve and relative performance shrink. That’s exactly what we see.
Any other example?
Take Olympic marathon times as an example. Men today run the race about 26 minutes faster than they did 80 years ago. But in 1932, the time difference between the man who won and the man who came in 20th was close to 40 minutes. Today that difference is well under 10 minutes.
What is the application to investing?
The application to investing is straightforward. As the market is filled with participants who are smart and have access to information and computing power, the variance of skill will decline. That means that stock price changes will be random—a random walk down Wall Street, as Burton Malkiel wrote—and those investors who beat the market can chalk up their success to luck. And the evidence shows that the variance in mutual fund returns has shrunk over the past 60 years, just as the paradox of skill would suggest. I want to be clear that I believe that differential skill in investing remains, and that I don’t believe that all results are from randomness. But there’s little doubt that markets are highly competitive and that the basic sketch of the paradox of skill applies.
How do you determine in the success of something be it a song, book or a business for that matter, how much of it is luck, how much of it is skill?
This is a fascinating question. In some fields, including sports and facets of business, we can answer that question reasonably well when the results are independent of one another. When the results depend on what happened before, the answer is much more complex because it’s very difficult to predict how events will unfold.
Could you explain through an example?
Let me try to give a concrete example with the popularity of music. A number of years ago, there was a wonderful experiment called MusicLab. The subjects thought the experiment was about musical taste, but it was really about understanding how hits happen. The subjects who came into the site saw 48 songs by unknown bands. They could listen to any song, rate it, and download it if they wanted to. Unbeknownst to the subjects, they were funneled into one of two conditions. Twenty percent went to the control condition, where they could listen, rate, and download but had no access to what anyone else did. This provided an objective measure of the quality of songs as social interaction was absent.
What about the other 80%?
The other 80 percent went into one of 8 social worlds. Initially, the conditions were the same as the control group, but in these cases the subjects could see what others before them had done. So social interaction was present, and by having eight social worlds the experiment effectively set up alternate universes. The results showed that social interaction had a huge influence on the outcomes. One song, for instance, was in the middle of the pack in the control condition, the #1 hit on one of the social worlds, and #40 in another social world. The researchers found that poorly rated songs in the control group rarely did well in the social worlds—failure was not hard to predict—but songs that were average or good had a wide range of outcomes. There was an inherent lack of predictability. I think I can make the statement ever more general: whenever you can assess a product or service across multiple dimensions, there is no objective way to say which is “best.”
What is the takeaway for investors?
The leap to investing is a small one. Investing, too, is an inherently social exercise. From time to time, investors get uniformly optimistic or pessimistic, pushing prices to extremes.
Was a book like Harry Potter inevitable as has often been suggested after the success of the book?
This is very related to our discussion before about hit songs. When what happens next depends on what happened before, which is often the case when social interaction is involved, predicting outcomes is inherently difficult. The MusicLab experiment, and even simpler simulations, indicate that Harry Potter’s success was not inevitable. This is very difficult to accept because now that we know that Harry Potter is wildly popular, we can conjure up many explanations for that success. But if you re-played the tape of the world, we would see a very different list of best sellers. The success of Harry Potter, or Star Wars, or the Mona Lisa, can best be explained as the result of a social process similar to any fad or fashion. In fact, one way to think about it is the process of disease spreading. Most diseases don’t spread widely because of a lack of interaction or virulence. But if the network is right and the interaction and virulence are sufficient, disease will propagate. The same is true for a product that is deemed successful through a social process.
And this applies to investing as well?
This applies to investing, too. Instead of considering how the popularity of Harry Potter, or an illness, spreads across a network you can think of investment ideas. Tops in markets are put in place when most investors are infected with bullishness, and bottoms are created by uniform bearishness. The common theme is the role of social process.
What about someone like Warren Buffett or for that matter Bill Miller were they just lucky, or was there a lot of skill as well?
Extreme success is, almost by definition, the combination of good skill and good luck. I think that applies to Buffett and Miller, and I think each man would concede as much. The important point is that neither skill nor luck, alone, is sufficient to launch anyone to the very top if it’s a field where luck helps shape outcomes. The problem is that our minds equate success with skill so we underestimate the role of randomness. This was one of Nassim Taleb’s points in Fooled by Randomness. All of that said, it is important to recognize that results in the short-term reflect a lot of randomness. Even skillful managers will slump, and unskillful managers will shine. But over the long haul, good process wins.
How do you explain the success of Facebook in lieu of the other social media sites like Orkut, Myspace, which did not survive?
Brian Arthur, an economist long affiliated with the Santa Fe Institute, likes to say, “of networks there shall be few.” His point is that there are battles for networks and standards, and predicting the winners from those battles is notoriously difficult. We saw a heated battle for search engines, including AltaVista, Yahoo, and Google. But the market tends to settle on one network, and the others drop to a very distant second. I’d say Facebook’s success is a combination of good skill, good timing, and good luck. I’d say the same for almost every successful company. The question is if we played the world over and over, would Facebook always be the obvious winner. I doubt that.
Would you say that when a CEO’s face is all over the newspapers and magazines like is the case with the CEO of Facebook , he has enjoyed good luck?
One of the most important business books ever written is The Halo Effect by Phil Rosenzweig. The idea is that when things are going well, we attribute that success to skill—there’s a halo effect. Conversely, when things are going poorly we attribute it to poor skill. This is often true for the same management of the same company over time. Rosenzweig offers Cisco as a specific example. So the answer is that great success, the kind that lands you on the covers of business magazines, almost always includes a very large dose of luck. And we’re not very good at parsing the sources of success.
You have also suggested that trying to understand the stock market by tuning into so called market experts is not the best way of understanding it. Why do you say that?
The best way to answer this is to argue that the stock market is a great example of a complex adaptive system. These systems have three features. First, they are made up of heterogeneous agents. In the stock market, these are investors with different information, analytical approaches, time horizons, etc. And these agents learn, which is why we call them adaptive. Second, the agents interact with one another, leading to a process called emergence. The interaction in the stock market is typically through an exchange. And, finally, we get a global system—the market itself.
So what’s the point you are trying to make?
Here’s a key point: There is no additivity in these systems. You can’t understand the whole simply by looking at the behaviors of the parts. Now this is in sharp contrast to other systems, where reductionism works. For example, an artisan could take apart my mechanical wristwatch and understand how each part contributes to the working of the watch. The same approach doesn’t work in complex adaptive systems.
Could you explain through an example?
Let me give you one of my favorite examples, that of an ant colony. If you study ants on the colony level, you’ll see that it’s robust, adaptive, follows a life cycle, etc. It’s arguably an organism on the colony level. But if you ask any individual ant what’s going on with the colony, they will have no clue. They operate solely with local information and local interaction. The behavior of the colony emerges from the interaction between the ants. Now it’s not hard to see that the stock market is similar. No individual has much of a clue of what’s going on at the market level. But this lack of understanding smacks right against our desire to have experts tell us what’s going on. The record of market forecasters has been studied, and the jury is in: they are very bad at it. So I recommend people listen to market experts for entertainment, not for elucidation.
How does the media influence investment decisions?
The media has a natural, and understandable, desire to find people who have views that are toward the extremes. Having someone on television explaining that this could happen, but then again it may be that, does not make for exciting viewing. Better is a market boomster, who says the market will skyrocket, or a market doomster, who sees the market plummeting.
Any example?
Phil Tetlock, a professor of psychology at the University of Pennsylvania, has done the best work I know of on expert prediction. He has found that experts are poor predictors in the realms of economic and political outcomes. But he makes two additional points worth mentioning. The first is that he found that hedgehogs, those people who tend to know one big thing, are worse predictors than foxes, those who know a little about a lot of things. So, strongly held views that are unyielding tend not to make for quality predictions in complex realms. Second, he found that the more media mentions a pundit had, the worse his or her predictions. This makes sense in the context of what the media are trying to achieve—interesting viewing. So the people you hear and see the most in the media are among the worst predictors.
Why do most people make poor investment decisions? I say that because most investors aren’t able to earn even the market rate of return?
People make poor investment decisions because they are human. We all come with mental software that tends to encourage us to buy after results have been good and to sell after results have been poor. So we are wired to buy high and sell low instead of buy low and sell high. We see this starkly in the analysis of time-weighted versus dollar-weighted returns for funds. The time-weighted return, which is what is typically reported, is simply the return for the fund over time. The dollar-weighted return calculates the return on each of the dollars invested. These two calculations can yield very different results for the same fund.
Could you explain that in some detail?
Say, for example, a fund starts with $100 and goes up 20% in year 1. The next year, it loses 10%. So the $100 invested at the beginning is worth $108 after two years and the time-weighted return is 3.9%. Now let’s say we start with the same $100 and first year results of 20%. Investors see this very good result, and pour an additional $200 into the fund. Now it is running $320—the original $120 plus the $200 invested. The fund then goes down 10%, causing $32 of losses. So the fund will still have the same time-weighted return, 3.9%. But now the fund will be worth $288, which means that in the aggregate investors put in $300—the original $100 plus $200 after year one—and lost $12. So the fund has positive time-weighted returns but negative dollar-weighted returns. The proclivity to buy high and sell low means that investors earn, on average, a dollar-weighted return that is only about 60% of the market’s return. Bad timing is very costly.
What is reversion to the mean?
Reversion to the mean occurs when an extreme outcome is followed by an outcome that has an expected value closer to the average. Let’s say you are a student whose true skill suggests you should score 80% on a test. If you are particularly lucky one day, you might score 90%. How are you expected to do for your next test? Closer to 80%. You are expected to revert to your mean, which means that your good luck is not expected to persist. This is a huge topic in investing, for precisely the reason we just discussed. Investors, rather than constantly considering reversion to the mean, tend to extrapolate. Good results are expected to lead to more good results. This is at the core of the dichotomy between time-weighted and dollar-weighted returns. I should add quickly that this phenomenon is not unique to individual investors. Institutional investors, people who are trained to think of such things, fall into the same trap.
In one of your papers you talk about a guy who manages his and his wife’s money. With his wife’s money he is very cautious and listens to what the experts have to say. With his own money his puts it in some investments and forgets about it. As you put it, threw it in the coffee can. And forgot about it. It so turned out that the coffee can approach did better. Can you take us through that example?
This was a case, told by Robert Kirby at Capital Guardian, from the 1950s. The husband managed his wife’s fund and followed closely the advice from the investment firm. The firm had a research department and did their best to preserve, and build, capital. It turns out that unbeknownst to anyone, the man used $5,000 of his own money to invest in the firm’s buy recommendations. He never sold anything and never traded, he just plopped the securities into a proverbial coffee can(those were the days of paper). The husband died suddenly and the wife then came back to the investment firm to combine their accounts. Everyone was surprised to see that the husband’s account was a good deal larger than his wife’s. The neglected portfolio fared much better than the tended one. It turns out that a large part of the portfolio’s success was attributable to an investment in Xerox.
So what was the lesson drawn?
Kirby drew a more basic lesson from the experience. Sometimes doing nothing is better than doing something. In most businesses, there is some relationship between activity and results. The more active you are, the better your results. Investing is one field where this isn’t true. Sometimes, doing nothing is the best thing. As Warren Buffett has said, “Inactivity strikes us as intelligent behavior.” As I mentioned before, there is lots of evidence that the decisions to buy and sell by individuals and institutions does as much, if not more, harm than good. I’m not saying you should buy and hold forever, but I am saying that buying cheap and holding for a long time tends to do better than guessing what asset class or manager is hot.
(The interview was originally published in the Daily News and Analysis(DNA) on June 4,2012. http://www.dnaindia.com/money/interview_people-should-listen-to-market-experts-for-entertainment-not-elucidation_1697709)
(Interviewer Kaul is a writer and can be reached at [email protected])