Image: President Barack Obama © Chip Somodevilla, Getty Images

President Barack Obama, who said his "one mandate" in a second term was to help middle class families, takes the oath of office with many barriers to raising most Americans' living standards.

Most Americans started this year with a cut in take-home pay as Congress let a temporary 2-percentage-point reduction in payroll taxes expire. Workers' leverage to gain wage increases will be limited for years by competition from the swollen ranks of jobless Americans as forecasters expect the unemployment rate to remain at or above 7% through 2014.

Even with bright spots such as signs of strength in housing and an energy boom that's lowering fuel costs for manufacturers, forecasters predict a slower expansion as federal tax increases and spending cuts crimp growth and demand for exports drops with a weakening global economy. While the U.S. economy advanced at a 3.1% rate in the third quarter, growth of just 2% is seen this year, according to the median estimate of economists surveyed by Bloomberg.

At the same time, Obama faces united Republican opposition to his agenda and pressure to slash federal spending.

"The president is still fundamentally on the defensive," said Damon Silvers, policy director for the AFL-CIO labor federation. "His fundamental task is to avoid being forced into austerity measures that will hurt the economic recovery or the capacity of our government to serve the people."

Falling incomes

Obama was confronted in his first term with an inherited recession that drove down most Americans' incomes. His second term starts with a fight to merely hold the line against the Republican House majority's push to cut benefits in entitlement programs such as Medicare and Social Security that many Americans rely on.

Looming battles over the budget, the legal debt limit and a round of automatic spending cuts scheduled for the end of February restrict Obama's leeway for other priorities, such as boosting infrastructure spending to stimulate economic growth.

The pieces of his economic agenda most likely to raise middle class living standards in the near term are "the parts that look like they're not going anywhere," said Larry Katz, a labor economics professor at Harvard University. These include plans for an infrastructure bank and tax breaks for new hires.

"Left by itself," Katz said, the economic recovery is likely to continue generating "tepid, small improvements" in middle-income workers' living standards "but not enough to really offset the poor performance since the late 1990s."

Long erosion

Middle class U.S. workers -- defined by one measure as households with annual incomes within 50% of the national median, or between about $25,700 and $77,000 -- have faced a four-decade-long erosion of living standards. It was interrupted by a rare period of rising earning power during the fast-growing economy of President Bill Clinton's second term, Katz said.

Real median household income soared to $54,932 in 1999 from $50,661 in 1996 based on constant 2011 dollars, an all-time high since the U.S. Census began reporting the data in 1967.

By contrast, median household income in November 2012 was $51,310 -- $3,850 lower than when Obama took office in January 2009, according to an analysis of census data by Sentier Research, an economic-consulting firm in Annapolis, Md.

In Clinton's second term, gross domestic product grew at an average 4.3% rate and the unemployment rate averaged 4.4%. That created a tight labor market, which provided workers leverage to press for higher wages.

Minimum wage

Clinton also won congressional passage of a two-step increase in the minimum wage, which took effect in 1996 and 1997, creating additional upward pressure on pay; the federal minimum wage hasn't gone up since 2009, when the final step of a wage increase signed by President George W. Bush took effect.

The gains of the late-1990s are eclipsed by decades of downward pressure on pay.

Median wages for men between 25 and 64 dropped 19% - - to $34,000 a year -- from 1971 to 2011, after accounting for inflation, according to an analysis by Michael Greenstone, a Massachusetts Institute of Technology economics professor and director of the Hamilton Project, a Washington research group affiliated with the Brookings Institution.

The impact of the decades-long slide in wages was initially cushioned in many families by the increasing number of women who went to work, and later by the home equity that families borrowed against during the run-up in housing prices before the real estate bubble burst.

More from Bloomberg: