The fiscal cliff may not be a real cliff, but jumping off it could be a catastrophe for the poor. Absent action from Congress and President Obama, come January 1, 2013, the Bush tax cuts, Obama’s payroll tax cut and extended unemployment insurance expire just as spending cuts from the sequester kick in. (To recap, in order to get Congress to lift the debt ceiling last year, President Obama formed a Congressional committee that was supposed to recommend ways to cut $1.5 trillion from the deficit. If it failed, “sequestration” would kick in—$1 trillion in automatic spending cuts split evenly between defense and non-defense spending, with Social Security, Medicaid and Medicare mostly protected, coinciding perfectly with the other expirations on January 1. The committee never came through, so now we’re facing down the cuts.) It’s not a pretty picture, although some have found silver linings. My Roosevelt Institute colleague Mark Schmitt is hopeful that real tax reform waits on the other side of the big leap. Jonathan Chait argues that the impact will be gradual enough that Obama can delay or cancel out most of it. Some Democrats, including Representative Peter Welch of Vermont and Howard Dean, think it’s worth going over the cliff in order to force Congress’s hand in getting the budgetary house in order.

These are all potential upsides of going over the fiscal cliff, but the downside for the country’s poorest would likely be very harsh. First, the budget cuts from sequestration will hit the poor incredibly hard—even if they will not represent a majority of the revenue raised. The term “non-defense discretionary spending” will warm few cockles of the heart. But it’s an incredibly important portion of the budget. Ethan Pollack of the Economic Policy Institute broke it down in the graph below:

As you can see, this money represents public investment: in education, transportation, children’s health, etc. And many of these programs directly impact the poor. Housing assistance, child care and education, nutrition assistance, home heating assistance and income security for the blind, disabled and aged all together make up 17 percent of this spending—more than the largest category in that chart. This money goes to job training, Title X family planning services and Head Start, among other things. It’s anything but discretionary for those who rely on these critical programs.

And they’ll get pummeled by these automatic cuts. According to a letter from the Obama administration released in July, the cuts will mean nearly 100,000 children losing Head Start services and the elimination of child care assistance for 80,000 others, for starters. That’s a lot of pain and suffering on the other side of the cliff.

But the impact on the poor doesn’t stop there. They’ll also be hurt by the tax cuts that are set to expire at the same time. Overall, the loss of the Bush tax cuts for income, capital gains and the estate tax will hit rich people harder, as the top 20 percent would see their effective tax rate increase an average 5.8 percentage points, while the bottom 20 percent would see an increase of only 3.7 points. But the Child Tax Credit and Earned Income Tax Credit, credits aimed at low- and moderate-income Americans, were expanded under the 2009 stimulus, with the former aimed in particular at low-income families. Those expansions will expire, too, if we tumble over the cliff.

This all means that while the richest will experience a larger increase in rates, the poor will actually feel the biggest hit to their bottom lines. The bottom 20 percent of Americans will see their taxes go up by an average of $209, reducing their after-tax income by nearly 2 percent. The top 40 percent, however, will only see their after-tax income dinged by .1 percent. That’s a lot of money to come from those already struggling to make ends meet.

There’s another group of vulnerable people who get smacked as well: the unemployed. As the recession began in June of 2008, President Bush signed into law the Emergency Unemployment Compensation program, which has since been reauthorized ten times. The National Employment Law Project reports that the unemployment rate is more than 40 percent higher now than when it was first enacted, and the percentage of the unemployed who have been out of work for more than six months has jumped by over 20 points. If Congress doesn’t reauthorize the EUC and the extended benefits expire along with everything else, 2 million workers currently collecting federal benefits will be cut off immediately and a million more will run out of state benefits by the end of the first quarter in 2013. That’s an actual cliff, not a gradual slope.

If a “grand bargain” is reached to avert some of this chaos that includes cuts to the social safety net, including the entitlements that are currently protected from the sequestration, poor people will of course still get hit hard. So that’s even more reason to favor solutions such as raising taxes on high income so that we can do what Katrina vanden Heuvel suggests will put us on a sustainable path: invest in our country. Revive manufacturing and lead in green jobs, invest in public education and affordable college and reign in Wall Street and CEOs’ excesses. It’s what voters want—they didn’t side with austerity to fix our problems. And it’s certainly what the poorest and most vulnerable among us need.

For more on why the fiscal cliff isn't really a cliff at all, check out "CliffsNotes for Washington."