Let’s start with what actually happened. Earlier this month, the American Enterprise Institute Club at Middlebury College invited Charles Murray to speak on campus. Murray has been the right’s version of a useful idiot for decades. In Losing Ground (1984) he’d argued that the social programs of the 1960s and ’70s aimed at improving the lot of minorities and the poor had done more harm than good. Ten years later, in The Bell Curve, Murray and co-author Richard J. Herrnstein claimed standardized-testing data proved that the differing fortunes of blacks and whites in America were explained not by racism or economic factors but by blacks’ being innately less intelligent. His 2012 book, Coming Apart, depicts the white working class as now falling prey to the same social pathologies he’d previously found among non-whites.

The event was co-sponsored by Middlebury’s political-science department. Allison Stanger, a political-science professor, agreed to act as Murray’s interlocutor “because while my students may know I am a Democrat, all of my courses are nonpartisan, and this was a chance to demonstrate publicly my commitment to a free and fair exchange of views.” Middlebury president Laurie Patton, who introduced Murray, said that, although she “profoundly disagree[s] with” many of his views, the “very premise of free speech on this campus is that the speaker has a right to be heard.”

Instead, as soon as Murray began to speak, a large proportion of the audience stood up, turned their backs to the podium, and began reading a statement that declared “these are not ideas that can be fairly debated,” followed by loud chanting. When it became clear the demonstration was not going to stop, Murray and Stanger repaired to another room, where, despite the noise from inside and outside of the auditorium, they had a conversation that was broadcast via livestream. Emerging from the building, Murray and Stanger found themselves surrounded by protesters who attempted to prevent them from leaving the campus. In the ensuing melee, Stanger’s hair was pulled, injuring her neck. Murray’s car was blocked, pounded on, and rocked back and forth, but as he finally pulled away from the crowd, he might well have found it difficult to credit his luck. Thanks to the crowd at Middlebury, a man justifiably regarded as a pariah has been transformed into a martyr for free speech.

So perhaps the first thing to note is that, despite the epidemic of hand-wringing, Murray was not in fact deprived of his free speech. Curious readers can find the full video of his hour-long conversation with Stanger on Middlebury’s website.

Which doesn’t excuse the attempt to silence him. As Bill McKibben pointed out, “Murray is a professional troll—‘Milo with a doctorate,’ as one observer described him.” In both cases, their campus tours are designed to provoke, not to enlighten or educate. But in responding to their provocations with violence, whether at Berkeley last month—when university authorities canceled a speech by Milo Yiannopoulos after protesters smashed windows and set fires—or at Middlebury, their opponents are playing right into their hands. And into a narrative that depicts college students in general—and liberal campuses in particular—as privileged enclaves filled with spoiled, out-of-touch, intolerant “snowflakes” more concerned with maintaining their own ideological cocoons than with making change in the real world.

Take a look at the coverage of Middlebury, not only in the right-wing press but also among liberal commentators. How many of those decrying the students’ response bothered to mention that this very campus also gave birth to 350.org—arguably the most serious, successful environmental group to emerge in this century? By ceding the high ground to Murray, and dissenting in a way that allowed them and their allies to be so easily caricatured and dismissed, the protesters were tactically foolish.

They were also deeply condescending to their fellow students, who are supposed to be developing the ability to think and argue and test their beliefs. Murray’s views may be odious, but students have a right to hear them and make up their own minds. You don’t need to be a free-speech absolutist to see that allowing a mob to veto unpopular ideas is a weapon that can all too easily be turned against progressive causes—especially in Trump’s America.

Yet it should also be acknowledged that in the current political climate increasing numbers of Americans, on and off campus, have reason to feel not merely devalued and insulted by Murray and his co-thinkers but actually under threat. Undocumented students and students with undocumented family members face the reality of deportation; students dependent on government aid that may be cut to pay for Trump’s military buildup face financial ruin. Muslim students and trans students face demonization and persecution.

Given those dangers, it is incumbent on institutions like Middlebury to exercise great care—not in whom they allow to speak; campuses especially should be open to all opinions—but in how controversial speakers are presented. By offering Murray not just a platform but also departmental sanction and a presidential introduction, Middlebury let down those students denigrated by his work. By allowing the inevitable, and justified, protests in response to degenerate from dissent to disruption to violence, the administration then let down everyone else.

All the student voices in the New York Times forum on the protest are worth hearing. Those torn between academic freedom and solidarity with classmates who felt threatened—not by Murray’s rhetoric but by its real-world consequences—are particularly heartbreaking. But we also need to heed the warning of senior Elizabeth Siyuan Lee, who criticized the university for restricting students to an essentially passive role in an event that lent Murray’s views “institutional legitimacy.” Lacking “a platform for legitimate discussion, it seems that students had few non-disruptive tools to get their voices heard.”

Despite our national mythology, free speech has never been free. The First Amendment was a dead letter—as the Supreme Court made clear in Davis v. Massachusetts, an 1897 ruling upholding a decision, written by Oliver Wendell Holmes, that city authorities have every right to ban public speakers from Boston Common—until 1939, when Mayor Frank Hague lost his fight to keep the CIO from handing out leaflets in Jersey City. In the interim, countless members of the Industrial Workers of the World were jailed in “free-speech fights” across the Northwest, while Emma Goldman, Margaret Sanger, and others were repeatedly, and successfully, prosecuted for disseminating information about birth control.

We gloss over this history at our peril. In detaching “free speech” from politics, and turning it into an icon of our own righteousness, Americans forget something we once knew—and that citizens of Rwanda, or Sarajevo, have had to learn the hard way: In deeply polarized societies, the gap from speech to violence becomes dangerously short.

Acknowledging that free speech always has a cost—and openly debating who should bear that cost—is the best, and perhaps the only way, to keep free speech from becoming a luxury good. The alternative is an America in which free speech, like access to health care, government-funded abortion, a secure old age, or clean air and water, is something that increasing numbers of us can no longer afford.