Every few weeks the government presents new statistics on how many new jobs have been created and how the unemployment rate is ticking downward. But dwarfing all those figures is the much more shadowy number of 94 million adults who aren’t working at all—and perhaps never did. So who isn’t working, and why isn’t work working for them?

The millions who don’t show up in the statistics are often overlooked or dismissed because they’re not employed or job seeking. But a fresh analysis by the Brookings Institution’s Hamilton Project reveals why a growing portion of “non-labor-force participants” are deciding work isn’t worth it. Why would anyone, after all, deliberately abandon the one activity that, above all, defines an American’s social status? These individuals seem to have opted out, but often it is the labor system that is turning away from their communities.

Their social profile is predictable and sadly familiar: Nonworkers generally skew older and poorer, particularly “prime age” men, aged 25 to 44. Many experience long-term disability; many, especially “prime age” women, are caring for others, rather than working for wages.

Researchers found that “More than 70 percent of labor force nonparticipants report that caregiving, disability, or early retirement kept them out of the labor force.” Despite not working, they’re subsisting, often barely, mostly on earned income from previous jobs or from another family member. A small percentage depend on public benefits or early-retirement income. Another 1.3 million have zero earnings and no fixed-income resources.

Though much of the media’s post-Trump spotlight has been trained on the grievances of “the white working class,” the real non-worker profile complicates gender stereotypes: Among non-workers, caregiving needs are a driving factor for multitudes more women than for men—36 percent of women cite caregiving as the primary reason for not formally working, compared with just 3 percent of men. The gap isn’t easily explained merely by cultural preference, much less “tradition,” since many many seem to be forgoing paid work to provide services to children, elders, or relatives with disabilities. The cost of employing professional home health aides or home care is out of range for many lower-income families. So non-working women might not bring home a weekly paycheck, but their days are still filled with work—just the unpaid, under-appreciated kind that has disproportionately always fallen upon women.

For non-working men, it’s a sad story in a different way. A higher proportion of non-working men are stuck in the lowest income bracket, compared with women. It’s not possible to discern from the data snapshot how many non-working men are considered “drop outs,” or how many were never in the workforce. But many reported having some earnings in the previous year, which suggests a temporary slump of unemployment followed by an eventual dropout.

Another recent study, by economist Alan Kruger, focused on so-called labor force “leavers,” draws a bleaker picture of the rupture and decline in workers’ life trajectories. The many so-called “discouraged” workers, who gave up job searching after many rejections, experience extraordinary rates of stress, social isolation and emotional misery—a collective pathology that experts often term “despair”. There are now signs that this long-term mental-health crisis is becoming a public health epidemic, as a new normal of prime-age adults, once considered the basis of the middle class, slide into frustration and social isolation.

Underlying the despair is chronic pain. Roughly half of prime-age nonworking men reportedly “take pain medication on a daily basis and that 40 percent report that pain prevents them from accepting a job.”

Extensive use of medication has also been linked to a rise in opioid dependency and overdose among older adults, which may soon be exacerbated by impending cutbacks to public health-care programs. According to Kruger’s analysis, the men see “disabilities and daily pain…as a barrier to regular employment,” and also report “the lowest levels of emotional well-being and life evaluation.”

Though the impact of the opioid epidemic on workforce participation is difficult to gauge, it’s clear that the cycle of disability and drugs aggravates economic strains; mass unemployment intertwined with long-term physical suffering is pushing millions of families deeper into economic decline and dysfunction.

Aside from aging non-working adults, who often face discrimination on the job market, many younger men are spending much of that time neither working nor seeking work. Much of the decrease is due to school enrollment and educational pursuits, but it appears a growing portion of non-working young men are engaging instead with leisure activities like video games.

Although many non-labor-force participants reside in households where other members are earning, safety-net programs like welfare and child support are also critical, particularly for non-working mothers, and since the Great Recession, deeply under-resourced. Overall, of non-working adults without other earned income, about one million men and 1.5 million women rely on public benefits.

Though it’s hard to see discern how all workforce dynamics interact with one another, the value of a strong social-welfare infrastructure remains crucial to keeping non-labor-force participants socially engaged. Government support can move people to a point where they are able to make the most of their potential, whatever their station in life.

State support for caregiving could simultaneously advance women’s economic recovery—for both their families and their wider communities. According to Hamilton Project researcher Ryan Nunn, “The reliance on informal caregiving may partially reflect a failure of the long-term care insurance market. Better-functioning long-term care insurance would help provide this funding.”

Yet caregiver services like childcare and home health care, also largely done by women, face massive understaffing, extremely low wages, and harsh working conditions. Expansion and equalization of paid caregiving labor would deliver gender and economic equity to both working women and those sidelined from work by domestic duties.

According to Nunn,“While it is difficult to speculate as to the labor market impacts of a more equitable division of caregiving work…better childcare and employment policies could support men and women who struggle to balance work and household responsibilities. For example, paid sick leave would be quite valuable for individuals who face intermittent caregiving responsibilities, allowing them to stay in the workforce”

There are many good and bad reasons people aren’t working; understanding why points us to economic hardships that we all are grappling with today, in and out of the labor force; there’s work to go around. But our economy isn’t letting us define for ourselves what it really means to work for a living.