Famed investor Doug Kass has decided to close his position in Berkshire Hathaway (BRK.A), suggesting that the stock's glory days are gone.

Take Monday's announcement by Warren Buffett that Berkshire had invested $10 billion in IBM (IBM), Kass wrote in a note. Berkshire's portfolio is so huge that it will have to make larger deals just to move the needle. He thinks the IBM deal shows that "more ordinary and plain vanilla" investments will be the core of Berkshire's portfolio strategy in the future.

"Berkshire's past growth and successes are one of its greatest enemies to future growth," Kass wrote. "Ever-larger investments/acquisit​ions are required to produce an impact and provide differentiated returns for shareholders."

And while Kass doesn't directly address Buffett's age or inevitable retirement, the change of leadership at Berkshire is clearly on his mind. Buffett has hired two investing gurus, Todd Combs and Ted Weschler, to help direct Berkshire's portfolio. Kass doesn't think they'll do as well as Buffett.

"They are mere mortals, and with the benefit of hindsight and history, Buffett was a true investing immortal!" Kass writes. "What is almost certain is that Buffett’s old black magic will not be easily duplicated by his appointees."

Kass' observations boil down to this: The bigger a company gets, the harder it is to grow. That certainly applies to investing portfolios as well.

He said he would buy Berkshire's class A shares at around $100,000 a share, but for now he sees better opportunities elsewhere. Berkshire shares traded Tuesday up slightly to $114,615.

Kass founded Seabreeze Partners Management, and started his investing career in 1972, according to his CNBC profile. He is a frequent investing commentator online and on television.