For many families in impoverished communities, their best hope for escaping poverty is to just move out of it. But often, the poverty follows them: They struggle to find a better neighborhood they can actually afford in crowded, expensive local housing markets. Today, with poverty and underfunded schools so intensely concentrated in isolated enclaves, the nationwide housing crisis is as much a crisis of segregation as a crisis of affordability.

The federal department of Housing and Urban Development has sought for years to help poor families relocate to lower-poverty areas through the Housing Choice program, which provides rental vouchers as a one-way ticket out to a healthier and stabler environment, resettling families in peaceful, integrated neighborhoods with more job and educational opportunities. Housing Choice vouchers are a major resource for cash-strapped public-housing authorities, currently supporting about 2.2 million households nationwide. Calculated according to income, the subsidies allow tenants generally to pay no more than 30 percent of income for rent. That could mean the difference between a roach-infested studio and a sunny single-family duplex with a $400 higher monthly rent. Those savings have a way of trickling down to the next generation, too: Research has linked a better home environment to healthier child development, less social distress, and greater economic stability. But as Congress weighs a modest expansion of the program, researchers have found that the Housing Choice recipients tend to face stiff barriers of stigma and implicit bias.

According to an extensive field study by the Urban Institute, many voucher holders are thwarted by landlords who simply won’t rent to them, regardless of the subsidy. Although some cities have laws that explicitly ban landlords from denying someone solely on the basis of being a voucher holder, researchers say subsurface biases still color first impressions and shape housing opportunity. It seems that people with vouchers are, ironically, perceived the same way landlords would view a bad credit check or a criminal record—a sign of a potentially troublesome tenant.

However, Urban Institute’s field testing in five cities—in which investigators conduct comparative inquiries of apartments, controlling for factors like income and race—show that the vouchers themselves might often perpetuate direct or indirect discrimination.

For the cities studied, it was challenging to find a landlord willing to even meet with a voucher holder. They combed through more than 341,000 rental ads for the five study sites, yielding a tiny average response rate of about one for every 39 ads screened. Across all five cities (Fort Worth, Los Angeles, Philadelphia, Newark, and Washington) people with vouchers were regularly turned down. The rates of denial varied, from about nearly four out of five queries in Fort Worth, three in four in Los Angeles, and two-thirds of Philadelphia landlords. Discrimination was predictably higher in housing markets with better (read higher-priced) conditions—places with better access to good schools, local transportation, and other community resources.

The one landlord who agrees to meet might seem like a godsend after 40 calls, but a foot in the door isn’t necessarily a reason to get your hopes up. In some cases when an in-person apartment showing was arranged, voucher holders were stood up by landlords at a higher rate. The findings reaffirm the trends revealed in the uptake of the program—vouchers often go unused by the households eligible to use them, indicating that it’s still hard to actually find a willing landlord, even with government support.

In reality, tenants with vouchers are generally more likely to make rent on time than those without, because they are enrolled in a regulated program that is overseen by a public agency. Ironically, this might be why bias barriers seem to ease slightly once the prospective tenant gets a foot in the door. Researchers report that when meeting face to face with either a voucher holder or non-voucher “tester,” “landlords who accepted vouchers appeared to scrutinize the control testers a bit more—for example, asking about income and employment more often.” That they expressed more wariness about the non-voucher applicant shows that not all landlords perceive stigma surrounding the program. According to Martha Galvez, co-author of the study, the landlords who are willing to meet with the voucher holders seemed to “understand [what] they’re accepting: they know they’re taking a low-income family that has this income bonus,” and they see it as an asset, not a drawback.

But while Housing Choice is meant to equalize opportunities to secure decent private housing, major gaps emerge in the friction of personal interactions that influence how a prospective tenant is seen: If they’re getting government help, does that make them more creditworthy, or less trustworthy, as tenants?

Remedying such deeply ingrained biases is a challenge for policy-makers, but policy makes a difference. Simply banning discrimination is one step: The renter tests in cities with anti-discrimination protections for bias against source of income saw somewhat lower denial rates.

Second, federal and local agencies can create more incentives for landlords in higher-income neighborhoods to participate in the Housing Choice program by streamlining the regulatory process. Meanwhile, the program could also be restructured to make administrative funds for public-housing authorities more flexible, to give local residents more support while searching for suitable apartments.

Another idea is to revamp the methodology for calculating the “fair-market rent” of a given area. The Center for Budget and Policy Priorities, for example, recommends calculating rental standards by ZIP Code, not metro area, allowing for a more accurate picture of what a single-bedroom costs in, say, the East Village versus Yonkers.

The barriers to voucher acceptance raise questions about how much the program really helps poor families access better-resourced neighborhoods. But Galvez says, “The program is working, it’s just not working as well as it could. And it’s not hitting the mark on this one very important outcome: helping people reach low-poverty, high-opportunity neighborhoods.”

The Housing Choice system offers just one route out of poverty, but many more are needed. A more comprehensive housing agenda would also invest to improve living conditions in high-poverty communities, so residents don’t have to move to seek better opportunities.

Still, the benefits of a good neighborhood environment should be redistributed in every way possible: whether by moving families into better neighborhoods or helping distressed communities develop so their residents can stay in place. The invisible barriers of Housing Choice remain one sign, however, that too many families still have no real choice at all about where they live, because no one gives them a chance.