(This is part 1 of a multipart series on common misconceptions about index funds. This article analyzes the relative performance of index funds vs. investing in individual stocks. Part 2 analyzes the cost advantages of index funds vs. stocks.)
I love getting feedback from articles I’ve written. Reading and responding to readers’ comments helps clarify my thinking.
Just recently I got a great comment on an article I wrote a while ago called Why I’ve switched from index funds to individual stocks. The comment was from Jonathan D Morgan. Here’s what he wrote:
Your perspective is sound, but your methodology is a bit flawed. If you understand the efficient market hypothesis then you understand that you can not beat the market. By selecting individual securities you are implicitly stating that you can select securities to beat the market. In reality your odds for beating or even matching the markets return are quite small, between 4-6% according to the SPIVA report. In the long run I think the research has proven that you will more than make up for the small fees you pay with the additional performance that comes from the diversification of index funds. You are also taking single stock risk, which is not mitigated unless you are purchasing the entire market. I think the index sampling methodology implemented by the index fund providers is worth the small fee of 0.01-0.05% depending on your balance. I pay a mere 0.02% for investing in index funds, and I have a PhD in finance and the resources to invest any way I choose. I choose low-cost index funds, and due to my balance I have access to institutional funds. Even I, with my vast knowledge of finance, am not confident in my ability to select winning stocks year in and year out.
A recent study by Longboard Asset Management looked at the formidable odds of beating the market from a different perspective. It compiled the returns of 3,000 stocks from 1983 to 2007. It shows that 39% of the stocks were unprofitable investments, 19% lost at least 75% of their value, 64% underperformed the market, and just 25% of the stocks were responsible for all of the market’s gains over the period. How confident are you that you can pick the right stocks for the long run? Good luck with your investing.
I really liked this comment – it had lots of thought behind it, it had some great numbers, it quoted some great studies, and I disagreed with just about everything in it.
Exaggerated claims
But before we dig into some of the fallacies about index funds, let me make something clear – index funds are great. I have a significant portion of my wealth in index funds. I believe index funds are the best investment vehicle for the vast majority of investors. Index funds are an easy “set and forget” investment. They are awesome for new investors or investors who want to regularly invest smallish sums of money. They are an easy way to capture the average return of the market.
Proponents of index funds aren’t content to stop there though. For some reason they feel the need to make additional claims that just aren’t backed up by basic statistics or common sense. I’m writing these articles to refute some of those claims.
I’m going to run through some of these arguments and the problems behind them, but before I do, let’s talk about what an index fund is.
What’s an index fund?
An index fund is just a mutual fund that owns either all or a representatively large sample of an index. And what’s an index? Just a big collection that represents some or all of a market. For an S&P 500 index fund that means the fund owns shares of all 500 of the largest companies in the US. The MSCI World Index is comprised of 1,653 of the largest companies in the world. To keep things simple, for the rest of this article we are going to talk about the S&P 500 index.
Most index funds are cap weighted, meaning they own more of the bigger companies and less of the smaller companies (where bigger and smaller are defined by the total capitalization of the company – that is, how much it would cost to buy the entire company). As of February, 2018, the largest company in the S&P 500 index is Apple, and it is 3.833967% of the index. The smallest company is News Corp, which represents just 0.007782% of the index.
If the index was evenly weighted then each company would be 1/500th, or .2% of the index. This means Apple is 19.2x overweighted and New Corp is .03x underweighted (compared to a proportional weighting of all 500 stocks in the index).
So what is the return for the index? It’s just the weighted average return of all of the constituent stocks that make up the index. Let’s look a simple example. Let’s assume we have an index of just 2 stocks that are evenly weighted:
We can calculate the return for this index very easily – it’s (4% * 50%) + (10% * 50%) = 7%.
And what if we changed the weighting to this (where B is now 75% of the index and A is just 25%)?
The return is (4% * 25%) + (10% * 75%) = 8.5%.
That’s all the math we need to dig into the misunderstandings about index funds.
Fallacy 1: Most investments underperform the index
Let’s start with the biggest fallacy first.
In Jonathan’s comment he referred to a recent study by Longboard Asset Management about the distribution of stock gains. I did a bit of research and tracked down the actual study. It finds, as Jonathan quoted, that 64% of stocks underperformed the Russell 3000 (the 3,000 largest companies in the US). 25% of the stocks were responsible for all of the market’s gains.
That finding makes sense intuitively to me. The stock market (and business in general) is something of a “winner takes all” affair. A small number of companies tend to dominate their industries (Intel with processors, Coke and Pepsi with beverages, Apple and Samsung with phones, Amazon with online retail, etc.) Those companies generate outsized returns.
For the purposes of this discussion I’ve put together a sample index that mirrors the findings in the Longboard Asset Management study. This simple index has 10 stocks and a total annual return of 10%.
The weighted return is just the annual return for a stock multiplied by its weight in the index. I included this because it makes it easy for us to see the impact of any given stock on the index average. For example, stock 7 had an annual return of 10% and it made up 10% of the index, so it added 1% to the annual gain of the index.
Let’s compare the findings from the study to the index above:
- 39% of stocks were unprofitable investments. In the index above, stocks 1-4 (40%) have a negative return.
- 18.5% of stocks lost at least 75% of their value. In the index above, stocks 1-2 (20%) lost at least 75% of their value.
- 64% of stocks underperformed the Russell 3000. In the index above, stocks 1-6 (60%) underperformed the index and stock 7 matched the index.
- 25% of stocks were responsible for all of the market’s gains. In the index above, stocks 8-10 (30%) were responsible for the index’s gains.
I think we can agree that this index pretty closely mirrors the findings from the study.
The problem is the conclusions drawn from the facts presented in the study. The statement “64% of stocks underperformed the Russell 3000” has been misinterpreted to mean that you have a 64% chance of underperforming the Russell 3000. That’s only true if you put all of your money into a single stock from the Russell 3000. Of course, that would be incredibly stupid, and I don’t know of a single financial advisor anywhere who would recommend that somebody put all of their money into a single stock.
Most people, of course, select a variety of stocks to create a portfolio. If we created a random portfolio of stocks, what should it return? Would we expect it to do better or worse than the index?
The average portfolio return must equal the index
I don’t want to get too far into the math here, so let’s make 2 broad observations.
First, in the example portfolio (as with the Russell 3000, according to the research), only a few stocks are responsible for most of the gains. In our example, just 1 stock was responsible for the entire positive return of the index. If you remove stock 10 from the index then the return goes from 10% to -11.1%. If you created a random portfolio from this index and your portfolio doesn’t have stock 10 then you’re guaranteed to have a negative return.
Second, we see that stocks 1-6 performed worse than the average, which means that if our random portfolio doesn’t include them then our random portfolio’s return would be better than the index.
What would we expect a random portfolio to return? Well, by definition, the return of an index is the weighted average of the stocks in the index, right? Similarly, the average of all possible random portfolios will also equal the return of the index. That is, if you took all of the portfolios made up of 2 stocks randomly chosen from the index, the average of those portfolios should be equal to the return from the index. Here’s an example:
If you have an index of 10 stocks there are 45 possible unique 2-stock portfolios. (Note: the number of unique portfolios made up of 2 stocks chosen from a 10 stock index is 10*9/2, since a portfolio with stock 1 and 2 is the same as a portfolio of stock 2 and 1. Similarly, if you had 20 stocks in the index there would be 20*19*18/2 = 3,420 unique 3-stock portfolios).
This table shows us that if you had a portfolio made up of stock 1 and 2 you’d receive a -85% return. If you had a portfolio made up of stock 4 and 8 you’d receive a 3% return.
The average of all of the portfolios above is 10%, which again, make sense. The average of all the 2-stock portfolio should be the same as the average of all the 1 stock portfolios, the average of all the 5 stock portfolios, etc. The average of all of those portfolios should be 10%.
Now let’s look at the above table again. How many portfolios actually underperformed the index return of 10%? 
I’ve highlighted all of the portfolios that underperformed the average. There are 23 such portfolios and their average return was a -23% return. There are 45 possible portfolios, so the mathematically astute among you will notice that 23 is more than half of 45 – it’s 51.1%, to be exact.
So there’s a 51.1% chance that a randomly chosen portfolio will underperform the index, and on average, an underperforming portfolio will underperform by a massive 33%!
Sounds like index funds for the win, right?
Not quite. Let’s look at it another way.
The MC 2999 index
What if you took the index and removed a single stock? Well, from the above calculation we’d guess that the new portfolio would have a 51.1% of being worse than the index, right?
Nope.
In our portfolio 60% of the stocks underperform the average. If you remove a stock whose return was less than the average, what happens to the average? It goes up!
So if you take the index and remove a single stock, there’s a 60% chance the new portfolio will beat the average. Here are the calculations:
If you remove any of stocks 1-6 you’ll see an improvement in the return of the portfolio. If you remove stock 7 you’ll see no change (since the return for stock 7 is the same as the average), and if you remove any of stocks 8, 9, or 10 you’ll reduce the returns.
The conclusion is clear – I am going to create a MC 2999 index. This will be the Russell 3000 index with a single, randomly selected stock removed. From the Longboard study we know that 64% of stocks in the stock market underperform the index average, which means that the MC 2999 will beat the market 64% of the time. I’ll be hailed as a genius.
Right?
Right…??
Of course not.
What are we overlooking?
How is it possible that a 2-stock portfolio is going to underperform the index 51.1% of the time, but a 9-stock index is going to outperform the index 60% of the time in this index and the MC 499 index will outperform the Russell 3000 64% of according to the Longboard Asset Management study?
It’s simple – the returns aren’t symmetric. In our 2-stock portfolio, 51.1% of the time we underperformed the index with an average -23% return (33% below the average). 48.9% of the time we outperformed the index with an average 65% return (55% higher than the average).
The 2-stock portfolio will underperform the index more often, but by small amounts. The 2-stock portfolio will outperform the index less often, but by larger amounts.
If you were to create a random 2-stock portfolio each year and roll your gains and losses from each year into the next year, your average portfolio return would eventually converge on the average index return. The more years, the stronger the convergence. For any given year, you’re slightly more likely to underperform than outperform, but the years you outperform you’ll outperform by more than you underperform.
A randomly chosen collection of stocks chosen from an index will underperform or outperform the index in any given year, but over time, the returns of the randomly chosen portfolio and the index will be the same.
Then why do most mutual funds underperform the index?
This is the easiest answer in this entire post. Most mutual funds underperform the index because of fees.
Does this mean that index funds are a superior investment vehicle? For some people, yes – index funds are the best investment vehicle.
But for somebody with a reasonably large portfolio, I believe you can effectively create your own index fund for less cost than an index fund. I’ll address that in the next post in this series.
Conclusion
Index funds are great, but index funds aren’t perfect. All too often, people draw invalid conclusions based on data. The reasons that most mutual funds underperform the index isn’t because of stock picking, it’s fees.
I’ll analyze and address other index fund fallacies in the rest of this series.
As a DGI guy beating the market is secondary.
Beating the market is great if you are into non dividend stocks. My goal is to get enough dividends to live off of, if I have extra at the end of the year I’ll invest that to keep more dividends coming in.
At the end of the day, the only thing that really matters is achieving your goals. For some people that means gains w/ minimal volatility. For others it means total return. For others it means beating the market. It sounds like you and I are interested in income.
“I really liked this comment – it had lots of thought behind it, it had some great numbers, it quoted some great studies, and I disagreed with just about everything in it.”
Thank you for the detailed post in response to my comment. I appreciate you taking the time to write such a thorough reply. However, you are fundamentally misunderstanding the efficient market hypothesis. You saying you disagree with it does not make it any less true.
First of all the concept that markets are efficient is not in question by any serious market participant, even the ones you mention. What is in question is the degree of market efficiency. Even Dr. Fama himself would concede that markets are not perfectly efficient but are efficient enough to validate the EMH. We have data and research going back to 1564 that supports the mathematical structure that leads to market efficiency. Additionally, the issue of behavioral anomalies in markets have also been addressed by Dr. Fama and others, and do not provide sufficient evidence to disprove EMH.
I think the issue here is possibly our objectives. If your objective is to maximize income, then you may very well be correct. I have done little research into the best way to structure a portfolio for income, because I find it to be a rather inefficient strategy. In truth, the best way to build a portfolio is through the lowest cost index funds possible, that are managed for total return, and then create your own dividend by selling a portion of your portfolio to meet whatever income need you have. Dividends signal to the investor that the company has nothing better to do with it, it is part of why Buffett does not pay a dividend, but this is a side point that I am willing to say we can agree to disagree on.
We dont need to argue about a great deal of the methodology you utilized here which I disagree with completely, rather I choose to focus on your concluding point that fees cause funds to underperform alone. This is not true. Fees are a contributor to underperformance, possibly the most important factor, but one can not dismiss the horrible record of active managers to pick stocks either. Active management is nothing more than a game of chance, a game of luck. Each year only a handful of stocks represent the majority of the return. If you or your manager happens to pick those stocks and holds them in higher weights than the market then you will outperform the market. If you or your manager fail to hold those stocks then you will underperform the market, at times considerably.
We can verify these theories by looking at the real life data found in the SPIVA reports. Over a 15 year horizon S&P found that “93.18% of large-cap managers, 94.40% of mid-cap managers, and 94.43% of small-cap managers failed to outperform the index” Let’s say you believe you are or have found a winning manager who can beat the index. Well then we should test their ability to repeat that performance consistently. The SPIVA consistency report does just that and found that funds performed worse than they would have been expected to perform by mere random chance. “only 4.03% of large-cap funds, 5.88% of mid-cap funds, and 5.75% of small-cap funds maintained top-half performance over five consecutive 12-month periods. Random expectations would suggest a repeat rate of 6.25%.”
I urge you to dig deeper into the research on this. Investment markets are efficient, and operate according to a zero sum game structure. Therefore, the odds are stacked against you to beat the market. As I stated, with your objective being income, you very well may be doing a better job and managing a portfolio with that objective. But to state that you disagree with EMH, is fine, but it is not supported by the research. Data and research should be driving investment decisions, not opinions.
Thank you again for your detailed response.
JDM
Well, first of all, it’s called the Efficient Market Hypothesis (EMH). That is, it’s an idea that has been proposed but does not yet have sufficient evidence to rise to the level of a scientific theory (which would mean that there is enough scientific evidence to be considered true). You simply saying it’s true doesn’t make it any more true.
Let’s take a simple example – there are 3 different forms of the EMH that have been proposed – the strong, semi-strong, and weak. The difference is what amount of non-public information you believe has been factored into prices. Given that all three are mutually exclusive, it’s impossible for all 3 to be correct. Which of the 3 variants do you believe has been proven?
Perhaps the issue is your definition of “efficient”. You claim that “…the concept that markets are efficient is not in question by any serious market participant…”. I believe the markets are “efficient” in that they quickly assimilate new information as it becomes available. But it’s a far cry from that to the conclusion that proponents of the EMH draw, which is that this efficiency means it’s impossible to consistently achieve higher returns than the market average.
I’m curious about the SPIVA research you cite. Did the research factor in fund expenses? Because if the average return is, say, 8%/year and the average fund has a .5% yearly fee, that would result in numbers very close to what you cite (.5% to 2.22% underperformance over a 5-year period). The compounding effect of these fees over 5 year will be very difficult to overcome for any active manager. If the SPIVA research didn’t add the fees back into the yearly performance then those numbers prove nothing about the relative importance of fees vs. stock picking for active managers.
I still don’t understand your conclusion that “the odds are stacked against you to beat the market”. A zero sum game means that your loss is somebody else’s gain, and vice versa. That doesn’t imply anything about anybody’s odds of success in the market.
Look at it a different way – what if you just selected 500 random stocks from the entire US stock market. Would you expect that collection of stocks to do better or worse than the S&P 500? If you believe that the random collection of 500 stocks will do as well as the S&P 500, then what’s so magical about the S&P 500? It’s just a collection of 500 stocks picked using a single criteria – market cap. If you believe the collection of 500 random stocks will do worse than the S&P 500 I’d be very interested in hearing your argument for why that would be.
Finally, what part of my methodology in this post do you not agree with?
Very interesting read. I have all of my pre-tax retirement savings in S&P 500 funds, mostly because they are the best options available in our work retirement vehicles. My after tax DGI account is comprised of individual securities because I am searching for better yield/yield growth than most indexes provide. However, I do have XSHD and SPHD in my after tax portfolio because they meet my investing objective for that vehicle and provide better sector diversification from the S&P 500. Good data!
Thanks. I think that index funds are almost always the best option in a 401k – you can virtually never purchase individual stocks, and a few index funds will allow you to capture the entire market’s performance.
Thanks very insightful writing and examples. I recently decided to use a hybrid approach and started investing with an ETF launched by the Motley fool (ticker TMFC) that invests in the top 100 of their stocks researched in their various newsletters. Hoping that will over the long haul give us a better than index performance.
That’s an interesting idea. For how long has that fund been around? Do they have any sort of performance date for it?
This is a new ETF however if you go to Fool100.com there they show the back testing of the same. Came up a couple of points higher than the S&P 500.
Since inception 221.17% S&P 142.76%
Annualized since inception 11.03% S&P 8.28%
Thanks very much, Amanda!
This analysis deals with the return side of the equation. What about the risk side? Any thoughts?
Good question. This begs the question – how do you define risk? The traditional way to measure risk in investing is via volatility, which I think is a terrible way to measure risk. See my post Volatility is not the same as risk for more details.
The reason most of mutual funds underperformed is not just because of fee. As average mitual fund annualized return is 3.5% lower than the S&P index. https://www.creditdonkey.com/average-mutual-fund-return.html. So it is more than only a fee.