Pennies on a stock chart © Tetra Images, Getty Images

Investors are piling into the shares of small, risky companies at the fastest clip on record, in search of investments that promise a chance of outsize returns.

The investors are buying up so-called penny stocks -- shares of mostly tiny companies that aren't listed on major U.S. exchanges -- at a pace that far eclipses the tech boom of the late 1990s. Those include firms that focus on areas from medical marijuana and biotechnology to fuel-cell development and precious-metals mining -- industries that are perceived by some investors as carrying strong growth potential.

Average monthly trading volume at OTC Markets Group (OTCM), which handles trading in shares that aren't listed on the New York Stock Exchange or Nasdaq Stock Market (NDAQ), has risen 40 percent this year in dollar terms from a year ago, to a record $23.5 billion.

The renewed interest in a market that used to be known as the pink sheets - because of the colored pieces of paper once used to record prices for unlisted stocks -- shows investors are ramping up risk in a bid to boost returns as U.S. stock indexes are hovering near highs and stock valuations have risen above historical norms.

Steve Templeton, 42 years old and a full-time day trader, said his winnings on unlisted medical-marijuana stocks allowed him to move to Tennessee, where there is no state capital-gains tax, from California earlier this year.

"I like things below three cents, because of the upside potential, and it limits the downside," Templeton said. "I buy twice what I want, and when it [doubles or triples], I sell half, and keep the rest."

One of his more successful investments this year has been Latteno Food (LATF), a maker of edible medical-marijuana products company based in Denver. He bought two million shares at $0.0008 in mid-January, and sold one million shares when the stock more than doubled to $0.002 less than two weeks later. Shares hit $0.0054 each in recent trading, more than five times the price he originally paid.

Many smaller companies have seen sharply higher trading volumes this year, including 2050 Motors (ZEGGD), a Las Vegas developer of carbon-fiber electric cars; MySkin (CNAB), a Newport Beach, Calif., provider of management services to spas; and GrowBlox Sciences (GBLX), an Orlando, Fla., medical-marijuana company.

Traffic on penny-stock trader forum InvestorsHub doubled between Christmas and last month, said Clem Chambers, chief executive of U.K.-based financial website ADVFN.com, which operates the forum. He said people who have been registered members for years, in some cases a decade or more, account for three-quarters of the increase.

"A lot of these people, who were looking from afar, are coming back," Chambers said. "It feels like somebody pulled a switch."

The rebound also comes as individual investors are showing signs of increased interest in stock trading in general. Discount brokers TD Ameritrade Holding (AMTD) and E*Trade Financial (ETFC) last month reported jumps in daily trading volume in the first quarter from the same period a year ago.

The rising volume in the tiniest of stocks is taking more investors into what is arguably the riskiest part of the stock market. These companies have less regulatory oversight than those traded on the exchanges, and their low prices mean that small price moves can quickly add up to big percentage moves.

In addition, penny stocks are often prime hunting grounds for scammers and "pump and dump" schemes. Stock promoters -- often masquerading as regular investors on chat boards -- tout a name, only to unload shares into a thinly traded market, taking profits for themselves but inflicting losses on other investors.

The Securities and Exchange Commission recently issued an investor alert warning of possible scams involving marijuana-related stocks, noting that "fraudsters often exploit the latest growth industry to lure investors with the promise of high returns." The SEC has halted trading in five marijuana-related stocks over the past two months because of fraud concerns.

While many of the shady "boiler room" brokerage call centers have been shut down by regulators, and although most of the volume is in legitimate companies with real profits, Chambers said it is still a market of "extreme risk."

"It's the closest you can get to pure gambling," Chambers said. "It's not all nightmarish companies, but the ones that capture the imagination are the most crazy ones."

Penny-stock investors also have broader market risks to consider: Liquidity in OTC markets dried up in 2001 as the tech bubble popped, and in 2009 following the financial crisis.

Some of the recent jump in OTC trading has come from shares of well-known companies, such as the delisted mortgage-finance firms Fannie Mae (FNMA) and Freddie Mac (FMCC) and large, foreign corporate titans that aren't listed on major U.S. exchanges such as Volkswagen AG (VLKAY), BNP Paribas SA (BNPQY) and Roche Holding AG (RHHBY). Excluding Fannie, Freddie and foreign shares, trading volumes are up 50 percent from a year ago.

Some investors say they have made money trading shares that could soon join the unlisted world: shares that are listed on an exchange but subject to possible delisting because of low prices.

Ryan Ung, a store manager in California in his 40s, realized a large profit by purchasing FuelCell Energy (FCEL) for less than $1 a share in early 2013, after the Nasdaq warned of a potential delisting for low share prices. That threat was removed in January 2013 after the shares rallied. Shares recently traded at $2.38, a little more than half this year's closing peak of $3.93, hit March 10.

Ung took a long break from trading penny stocks after the stock-market collapse in 2008 but started following the market again in early 2013, after an online newsletter on alternative-fuel stocks rekindled his interest. "I wanted to find something small" with a high profit potential, Ung said.

Not all penny-stock trades work out so nicely. Templeton bought Citadel EFT (CDFT), a credit-card-processing-services company with interest in the medical-marijuana industry, at 82 cents a share. The stock was halted by the Securities and Exchange Commission at 68 cents after the market closed March 20 because of questions regarding the accuracy of several public disclosures.

After the halt was lifted following the company's response to the questions, the stock closed at eight cents a share on April 4, and was recently changing hands at two cents.

Click here to become a fan of MSN Money on Facebook

More from The Wall Street Journal