Businessman in handcuffs © Siri Stafford, Getty Images

The cramped office tucked behind an auto-parts supplier in a Boston suburb was an unlikely setting for a hedge-fund manager with a $3 billion New York firm.

But John Kelly, a middle-aged man partial to pleated navy slacks and a pink button-down shirt, made clear to visitors that he had good reason for operating his firm, SeaFin Capital, far from the prying eyes of his Wall Street bosses: He was running a scam and looking for willing conspirators. What he didn't tell guests, for obvious reasons, was that his real name was John Keelan and he was an agent with the Federal Bureau of Investigation.

The office, business cards and SeaFin Capital website -- even a squishy plastic orca with the firm's logo on the bottom -- were props in Operation Pennypincher, one of the biggest investment-fraud stings ever mounted by the U.S. government. The target was the murky world of penny stocks, a historical haven for con men and hustlers that the FBI says is "rife with fraud."

Microcaps are tiny companies, typically with share prices of a penny or less, and often trade on quotation systems that don't set minimum financial standards like the major stock exchanges do. Investors are buying such stocks at a record pace, recent trading data show. Even though the companies are small, scams involving them can run into tens of millions of dollars.

To expose these and other Wall Street frauds, government officials increasingly are turning to law-enforcement tools that for years were reserved for organized criminals such as drug dealers and the mafia. Wiretaps have proved the smoking gun in a string of recent insider-trading convictions, though white-collar stings have more of a mixed track-record in court.

Operation Pennypincher lasted more than a year and still has cases winding through the courts. To date, 22 people have been charged criminally, 18 of whom have pleaded guilty or been convicted after trial on various counts of fraud. The latest charges were filed in February.

The Wall Street Journal has pieced together this account based on thousands of pages of court documents, surveillance videos shown at trial and discussions with people familiar with the operation.

The sting began with a series of videotaped meetings starting in 2010. Keelan -- posing as Kelly -- found middlemen who would help him reel in executives of penny-stock companies willing to pay a huge kickback in return for an investment of up to $5 million by SeaFin.

The agent stressed that his employer was to be kept in the dark about the backroom deals, which effectively involved stealing millions of dollars from the fund's investors, prosecutors said. "My hedge fund doesn't know . . . and they don't need to know," Kelly said in one of the meetings.

The scam wouldn't raise red flags at the hedge fund because the firm would get stock certificates for the full amounts invested, not reflecting the kickbacks. In addition, the fund paid inflated prices for the shares, which typically were restricted shares that weren't traded publicly. The agent made it clear he didn't expect any of the investments to work out.

One of the conspirators, a 71-year-old former stockbroker from Lincoln, Rhode Island, named Edward Henderson, accepted a $12,650 payment for bringing in a handful of executives even though he feared Kelly was so brazen about his wrongdoing that the whole thing "could be an FBI sting."

Yet on May 24, 2011, he drove to an upscale Burlington, Massachusetts, restaurant to meet Kelly for lunch and discuss more deals. Henderson was busted in the parking lot. ("I never got the lunch," he later told a court.)

Faced with the stark choice of cooperating or facing arrest, a "horrified" Henderson a few days later admitted his guilt -- even confessing to past, undetected insider trading -- and told the FBI he would help with its sting.

One of the first phone calls he recorded was with James Prange, a 63-year-old from Greenbush, Wisconsin, with a track record in business that included an attempt to open a bratwurst "Hall of Flame" celebrating the sausage. Prange also helped penny-stock companies raise money, a job his lawyer later described as "repetitive, frequently frustrating and disappointing."

Prange wasn't on the FBI's target list, according to people close to the sting. But in the insular penny-stock world, word had spread rapidly about the Boston hedge-fund manager with a multimillion-dollar checkbook that could be opened by kickbacks.

In calls some weeks later, Henderson -- by now wearing a wire -- told Prange how the hedge-fund scam worked, according to court documents. The penny-stock promoter "jumped at the opportunity" to take part, prosecutors said, agreeing to fly to Boston with executives from companies he represented to meet the corrupt fund manager.

FBI agents had worried they had made the terms of the scam so clearly illegal, to try to avoid claims they were unfairly entrapping people, that no one would be caught, according to people close to the operation. In fact, "we should have set up a turnstile," one of the people said.

Prange's appeal of his later conviction for fraud likens the sting's targeting of "down and out companies in desperate need of capital" to buying expensive drugs for "poor, homeless drug addicts."

Prange was introduced to the purported hedge-fund manager at a meeting on July 22, 2011, with Henderson in attendance. Kelly told Prange he could arrange for SeaFin to invest up to $5 million per company, irrespective of its prospects, in exchange for secretly being handed back half of the money. The executives, in turn, could do whatever they liked with what was left of the cash.

As a "finder," Prange would get a 10 percent share of the $2.5 million kickback for each company he brought in to the scheme, the agent said.

"It's basically quarter a million a deal for you," Kelly told Prange. "I don't know . . . if that squares with how you were thinking?" Prange nodded in agreement: "Sure."

Pinching pennies: infographic

The agent also explained how the kickbacks would be covered up using fake invoices paid to a sham consulting company that was a front for him. The consulting company, called Watersedge Group LLC, was another FBI invention.

During meetings with potential conspirators, an FBI agent would call Keelan at his desk to make him seem legitimate.

A month later, Prange brought John C. Jordan, 62, from Cameron Park, California, to meet the agent -- one of three chief executives he introduced to the scheme, according to prosecutors. The meeting lasted more than two hours, with Jordan talking at length about his animal feed company, Vida Life International. Originally from "the capital of fishmeal country" in Peru, Jordan is "passionate about fertilizer, fishmeal and fish oil," according to his lawyer.

The secret tape of the meeting showed the agent setting out his terms: "I'm expensive . . . I get a 50 percent kickback on whatever I invest."

Jordan later told the agent: "Your fee is fine because you're definitely bailing me out."

A week after the meeting, the FBI sent $32,000 to Vida Life. The scheme involved payments made in escalating slices -- an arrangement supposedly designed to build trust, but actually created with the FBI's limited finances in mind, said people close to the operation.

Within days, Jordan had wired the $16,000 kickback to the fake consulting company. He used much of the remaining money for payments to himself, his family and his business partner, prosecutors said.

Another chief executive that Prange introduced to the agent, Karen Person from Las Vegas, was last year sentenced to 30 months in prison after pleading guilty to securities fraud. Her lawyer didn't immediately respond to a request for comment.

Last year, Henderson was rewarded for his guilty plea and extensive cooperation with a sentence of one-year probation. A federal jury in Boston in May last year convicted Prange and Jordan of fraud. Each man is now serving a 30-month sentence at a low-security prison. Lawyers representing Prange and Jordan declined to comment, as the two men are appealing their convictions. Neither Henderson nor his lawyer responded to requests for comment.

Prange's incarceration followed a dramatic arrest, according to evidence at his trial. At 6 a.m. on Dec. 1, 2011, three FBI agents and two Sheboygan County deputies came "storming in the driveway" of Prange's Wisconsin farm home, "everybody wearing their gear," his lawyer said.

The gun-toting agents found the penny-stock promoter, his lawyer added, "cowering down in a closet, scared out of his mind."

Click here to become a fan of MSN Money on Facebook

More from the Wall Street Journal