Stock market © Zurbar, age fotostock

When the price of oil mysteriously plummeted $4 last week with no news to move it, seasoned market pros figured they knew what had happened.

The ghosts in the machine were back at work.

Those ghosts are the bots that now carry out more than half of all market trades, using rapid-fire strategies known as high-frequency trading.

The bots and other high-tech, automated trading systems are creating train wrecks more and more often, at a big cost to individual investors like you and me. (They're also feeding a crisis of confidence in markets; read "8 ways investing is broken.")

It's time to rein in these bots before they destroy our portfolios. If you think that warning is too dire, consider the following scary examples:

  • The flash crash of May 2010 wiped out a trillion dollars' worth of wealth in a matter of minutes, as markets mysteriously plunged and then bounced back like a bungee jumper.
  • The Facebook (FB) initial public offering fiasco left investors in the dark for hours on whether they had bought or sold shares, causing them to double-enter orders and make other costly errors.
  • The early August blowup at Knight Capital Group (KCG) mistakenly sent thousands of orders into the market, causing dozens of stocks to tank.

Image: Michael Brush

Michael Brush

Perhaps more frightening: While high-profile incidents like these make headlines, smaller incidents happen all the time. In the past year, there have been mini-crashes in more than 2,000 stocks, according to Nanex, a market data company. Disturbingly, Chicago Fed researchers recently found that all exchanges, and two-thirds of trading shops they interviewed for a study on tech risks in the market, had experienced errant algorithms -- bots gone wild.

Since regulators haven't taken any serious steps toward fixing these problems, more disasters are bound to happen. "U.S. equity markets are in dire straits," says David Lauer, of the lobbying group Better Markets. "It is simply a matter of time before we have another catastrophe of the same magnitude or worse than the flash crash. We are truly in a crisis."

If anyone knows, it's Lauer. He used to be a high-frequency trading analyst for a private hedge fund.

How this all costs you

Wall Street bots are basically supercomputers armed with algorithms developed by computer geniuses to execute trades in milliseconds, often without human intervention. And they can make bad moves and bad calls.

Worse, even when broken bots aren't causing meltdowns, their tricks are costing you in a variety of ways. High-frequency traders sometimes deliberately flood quote systems with orchestrated trades to delay feeds to the rest of us, so they can take advantage of prices they know about through exclusive quote feeds, before the rest of us. One study concludes that this practice, called quote stuffing, happened with 74% of stocks in 2010.

But wait, you say; I'm not a day trader and don't even trade often, so how does this hurt me?

In fact, bots gone wild -- and the market damage caused by bots even when they are working well -- carry huge risks to individual investors. I group the costs in three categories:

Direct costs to stock investors. During the May 2010 flash crash, more than 300 stocks fell more than 60% in an instant. Investors who panicked, or who had stop-loss orders (defensive orders that automatically sell a stock if it falls by a certain amount) in place, wound up unloading stocks at big discounts. The stocks promptly recovered, but the losses were locked in.

The same thing happened when the Knight Capital trading system went awry, flooding the market with orders that hit many stock prices.

Even when meltdowns aren't playing out, high-speed trading puts us at a disadvantage as individual investors, because exchanges let traders use new hybrid order types to get first in line in the market. The hybrid orders have esoteric names like "hidden midpoint pegs," "hide-and-slide" and "post-only." "They have to get to the top of the book, because that is where speed is advantage," says Joe Saluzzi, of Themis Trading, which goes up against HFT traders when executing trades for institutional investors.

The direct costs to pension funds and mutual funds. Think you're safe because you own mutual funds instead of stocks? Think again, because HFT traders are gaming the system against the traders for your funds. One trick: High-frequency traders flash potential prices to gauge market interest, says Andrew Brooks, the head of U.S. equity trading at T. Rowe Price (TROW).

In fact, HFT systems cancel more than 90% of their orders, Brooks says, and for good reason: They were never meant to be executed. Those orders are merely attempts to sniff out when a mutual fund has a big position to move, so they can trade ahead of it. The result is that mutual funds pay more for stocks they're buying and get less for those they sell. "There is a lot of money being taken out of the pockets of long-term investors," says Brooks.

A Morgan Stanley study confirms that orders from mutual funds and pension funds are having a much larger impact on stock prices now compared with five years ago -- a sign that high-frequency traders are tracking and trading on those moves.

You might write this off as smart people using technology to read the hands of other market players, just as you might use body language to figure out how low a car dealer can go in price. But the playing field isn't level, because high-frequency traders have special advantages here that the rest of us do not enjoy. They have data feeds from exchanges that give them quotes faster than anyone else can get them. They even set up their computers physically close to exchanges to get another tiny speed advantage. "This creates an uneven playing field that favors those who can and will pay for it," says Saluzzi, a co-author of "Broken Markets: How High Frequency Trading and Predatory Practices on Wall Street are Destroying Investor Confidence and Your Portfolio," a book that outlines the problem.

The cost to our economy. Of course, people aren't dumb. Investors have noticed these problems, and they're backing away from the stock market. "As we talk with our clients, there is a growing distrust of the casino-like environment that the marketplace has developed over the past decade," says Brooks. This is no doubt part of the reason that investors have pulled more than $280 billion from stock mutual funds since the 2010 flash crash. "Investors have been fleeing the stock market in droves," says Lauer. Private companies are doing the same, going public much less often. Both trends hurt our economy, since it means that fewer companies are using the stock market the way it was intended: as a place to raise money to fund growth and create jobs, and then reward those who invested. Instead, the market becomes ever more like a game, with nothing real created.