By James Brumley

It feels a little unnatural to criticize the icon of value investing, particularly when he's got a superior long-term track record. On the other hand, when arguably the wisest fund manager on the planet starts to trail the market's average performance on a regular basis, even the most patient of investors is going to start second-guessing the guy making the picks.

And that's exactly what's happening now with Warren Buffett's Berkshire Hathaway (BRK.A, BRK.B) investors.

Between April 2011 and April of this year, Berkshire shares lost 2.4% of their value, while the market gained 2.8%. In and of itself, that disparity isn't a huge deal. But when it's the tail end of three years' worth of lagging performance -- during which the fund advanced 32% while the market gained 60% -- shareholders have to start asking some tough questions.

The most important of those questions also are the simplest: Which stocks aren't cutting the mustard, and what went wrong? The answers might surprise you.

Berkshire's biggest losers

The drag any particular stock has had on the portfolio really depends on the time frame in question. Some have lagged for three years, and some for only one. It's even trickier than that, though, as the portfolio always is changing, making it tough to pinpoint what lost what when. As for the biggest drags we can identify, though ...

• Bank of America (BAC): Berkshire doesn't own it any longer, but it was a fairly big holding back in 2010 that he sold in the fourth quarter of that year. Thing is, the stock had lost more than a third of its peak value from 2010 by the time fourth quarter rolled around. That said, note that Buffett still has a Bank of America position via preferred shares, which is doling out a pretty sweet 6% yield. The preferreds also are convertible into common shares at $7.14, but for the time being it's acting like fixed income.
• Bank of New York Mellon (BK): This was purchased some time in the third quarter of 2010, which initially proved to be good timing -- the stock bounced back from $25 then to a peak of $32 by the beginning of 2011. Unfortunately, it hit sub-$18 levels by October of last year, and he still owned most of it then.
• Wal-Mart (WMT): Buffett has had it for the whole three-year span. Though it was a minor position (about 3% of the whole portfolio), it still made no meaningful net progress between early 2009 and mid-2011, which didn't help his overall results.
• ConocoPhillips (COP): This probably is the biggest drag on the portfolio for the past year, as well as for the past three years. The stock has lost 31% for the past 12 months and has gained just about 19% for the past three years. Buffett acknowledged that he paid too much for ConocoPhillips.

Of those four names, Conoco probably would be categorized as the biggest letdown, even by Buffett himself.

Funny thing about all four of those stocks, though, even as Berkshire Hathaway's performance has lagged, it's not like any of those picks have been outright disasters. Rather, the mediocrity and sub-mediocrity from Berkshire since 2009 actually has been spread across most of its holdings, of all sizes, and in most sectors.

And there's something else. See any missing names? Don't forget that Berkshire Hathaway also outright owns Lubrizol, Burlington Santa Fe, Shaw, Johns Manville, Acme Brick and MiTek, just to name a few. It also owned a big bond position in Energy Future Holdings last year. Since these names aren't publicly traded, though, there's a bit of a challenge in determining the intrinsic value they represent to the whole company.

It's what you can't see that matters most         

It's a reality that's rarely discussed about Berkshire's structure, but it's actually a fund (sort of) made up of both publicly traded companies as well as privately held ones. We can wrap our hands around the publicly traded ones pretty well. It's the privately held names here, however, that might be the biggest reason for the three-year struggle.

The construction names owned by the conglomerate illustrate the idea. Shaw (carpet), Johns Manville (insulation), Acme Brick (brick and tile) and MiTek (building products) together earned $1.8 billion in 2006. Last year, they collectively earned $513 million. How did this impact the intrinsic value of Berkshire shares? Great question. Their intrinsic value is only a theoretical one based on a multiple of earnings, unlike the publicly traded companies owned by the portfolio, which trade at finite values.

Over the past couple of years, Berkshire Hathaway also wrote down about $1.4 billion worth of the $2 billion investment in Energy Future Holdings. It was on the books, but buried in a mountain of accounting statements, and obscured by all the publicly traded stocks Berkshire holds. But clearly that markdown had an impact on the overall company.

So to answer the initial question, the reason Berkshire Hathaway has performed so tepidly since 2009 might have more to do with the companies you can't dissect, and less to do with the ones you can. Like any corporation, "what went wrong" was weak results. You just had to dig deep to glean that.

Bottom line

Warren Buffett remains a great investor, but he's not infallible.

What's interesting about the past three years is that Buffett paid dearly for a couple of the privately held companies that might be failing Berkshire now. Namely, he paid 31% more than the going price for the 2010 acquisition of railroad Burlington Northern Santa Fe, and offered a 28% premium for Lubrizol last year. And given the bids, odds are those are not the only businesses he overpaid for -- they and Conoco are just the ones we readily recall.

Simply put, when the man himself takes "value" out of value investing, lagging performance shouldn't be a surprise.

As of this writing, James Brumley did not hold a position in any of the aforementioned securities.

Related Articles

• 5 boring stocks, but 5 sexy yields
• The REAL story of "Sell in May and go away"
• 'Top Stocks' pick becomes a total disaster