How much will families spend on gifts this holiday? That's a key question for retailers heading into the critical fourth quarter.

The news isn't good. The wealthiest people in the country -- those with a net worth of $800,000 or more -- say they won't spend as much on gifts this year, according to the American Affluence Research Center.

The survey found that this group expects to spend about $2,370, on average, down from $2,399 last year (that's the total spent by all adults in the household). Only 3% said they plan to spend more, while 28% intend to spend less, The Wall Street Journal reported. And 12% said they won't buy gifts at all.

"High-end retailers are unlikely to take much cheer in the finding, since so much of our consumer economy is now built on increased spending by the wealthy," writes Robert Frank.

This may become a point of concern for Coach (COH), Nordstrom (JWN) and other retailers focused on affluent shoppers. With less money to spend, the wealthiest may scale down this holiday to such retailers as Macy's (M) and Target (TGT).

It's not supposed to be this way, Frank notes. The wealthy should have been ready to spend again this holiday. "The millionaire population is back up," he adds. "Stocks are well off their lows of 2009. And high-end incomes have held up fairly well -- especially compared to the rest of the country."