Now that the fiscal cliff is in the rearview mirror with a deal that raises income and investment taxes on the wealthiest Americans, there's one tax hike that most U.S. workers will not escape.

On the final day of 2012, the payroll tax for Social Security reverted to its normal rate of 6.2% -- up from the 4.2% it's been since 2011. The tax had been temporarily reduced to give U.S. households extra spending money to help stimulate the economy.

For a household grossing $50,000 a year, the return to the old level will cut take-home pay by $1,000 a year, or $19 a week. For someone making $30,000 in pretax income, the higher tax means $600 less a year.

How much more will you pay? The Wall Street Journal provides a calculator, or you can consult this handy chart. Keep in mind that the payroll tax applies only to the first $113,700 of income.

Extending the payroll tax holiday wasn't even part of the discussion as Congress narrowly avoided the fiscal cliff with a deal that raises income and investment taxes on those making $400,000 a year ($450,000 for couples).

However, by allowing Social Security taxes to return to their previous level, Congress can't say the middle class was spared.

"It's a huge hit," Joel Naroff, president of Naroff Economic Advisors, told The Associated Press. "It hits people whether they're making $10,000 or they're making $2 million. It doesn't matter who you are . . . . The lower your income, the more of your income you're (spending). So if your taxes go up, it's going to come out of your spending."

The Washington Post says this will be the first tax increase nearly half of Americans have seen in their paychecks (other than when your income grows and you move into a higher tax bracket).

The Post added:

"'We haven't seen broad-based individual tax increases at the federal level in the last 30 years,' said Owen Zidar, an economist at the University of California. 'In the 1960s through the 1980s, payroll tax increases affected most taxpayers, but the vast majority of broad-based tax changes have been cuts rather than increases.'"

On the other hand, Social Security has to be funded. That money normally comes from a 12.4% payroll tax, half paid by you and half by your employer. Only employees got the reduced 4.2% rate over the past two years.

The estimated $215 billion workers didn't pay to Social Security for 2011-12 was replaced with money borrowed by the federal government, which added to the national debt.

Will the economy suffer because the tax holiday expired? Maybe. Maybe not. Says the Post:

"Economists say the expiration of the tax cut will be a major drag on the economy this year. Estimates suggest it could cost between 500,000 and 1 million jobs, leaving the unemployment about 0.4 percentage points higher than it otherwise would be."

However, Joseph Rosenberg, a research associate at the Urban/Brookings Tax Policy Center, told American Public Media's Marketplace that if the Bush tax cuts are extended (and they were for the vast majority of folks), "then the impact of the expiration of the payroll tax is not likely to have a significant economic impact."

It is the wealthy who will be seeing the biggest tax increase as a result of the fiscal cliff deal. Says the AP:

"For 2013, households making between $500,000 and $1 million would get an average tax increase of $14,812, according to the Tax Policy Center analysis. Households making more than $1 million would get an average tax increase of $170,341."

Will you miss the extra $19 or so a week you took home during the payroll tax holiday?

More on MSN Money:

• Deducting medical bills to get harder in 2013
• Late-night deal averts fiscal cliff
• More bruising budget battles are coming
• Smart Spending on the go: Get our app for Android or iPhone
• 6 changes to Social Security in 2013
• A bigger Social Security check?