When PEN decided to award the first PEN/Toni and James C. Goodale Freedom of Expression Courage Award to the French satirical magazine Charlie Hebdo, the group surely thought it was honoring bravery in defense of free speech. This was a magazine that kept publishing after its offices were firebombed by Islamists in 2011, and kept publishing after nine staffers were murdered by Islamists in January. Compare that to, say, Yale University Press, which dropped the illustrations for Jytte Klausen’s book about the Danish Muhammad cartoons after the book’s first printing, or Random House, which canceled publication of Sherry Jones’s The Jewel of Medina, a novel about Muhammad’s wife Aisha. Both publishers cited fears of violence. Those fears were not irrational: The home of the publisher who picked up Jones’s novel was firebombed—and the book was dropped. Violence works.

Courage is not enough, though, for the six novelists—Francine Prose, Teju Cole, Michael Ondaatje, Peter Carey, Rachel Kushner, and Taiye Selasi—who have withdrawn from participation in PEN’s awards gala on May 5. Kushner criticized the magazine’s “cultural intolerance” and support for “a kind of forced secular view.” Writing in The New Yorker after the murders, Cole condemned Charlie Hebdo for racism and Islamophobia. Carey had trouble with the concept of freedom of speech being restricted by anything but government. Salman Rushdie then called the writers “pussies” on Twitter, which didn’t help.

Charlie Hebdo’s work is not important,” Francine Prose told me. “It’s not interesting.” She said she was offended by Charlie’s crude cartoons of the Prophet and its mockery of the religion of France’s marginalized Muslim community: “It’s a racist publication. Let’s not beat about the bush.” She compared the magazine’s Muslim caricatures to Goebbels’s anti-Semitic propaganda: “I don’t see a difference, really. It’s the same big noses and thick lips.” She pointed out that many dozens of Mexican and Russian journalists had been killed for reporting on government corruption in their countries. Why not honor them? (The imprisoned Azerbaijani journalist Khadija Ismayilova is receiving the night’s other award.)

I’ve known Francine since we were in college, and admire her and her writing enormously. I agree with her that there’s a distinction between supporting freedom of speech and honoring the speech itself. It’s probably safe to say that if PEN believed Charlie was the Völkischer Beobachter of our day, it wouldn’t be giving the magazine an award, no matter how many of its editors had been massacred. I don’t agree that the Muhammad cartoons are in a different key than the magazine’s rude caricatures of the pope or Hasidic rabbis or the Virgin Mary just after being raped by the three kings, but maybe that’s in the eye of the beholder. In any case, Charlie is a small satirical magazine run by aging ’60s leftists who spend the vast bulk of their column inches attacking the National Front and other French conservatives, with frequent jabs at the Catholic Church. Those immersed in French cartoon culture have pointed out that the offensive drawings circulating on the Internet are, in context, the opposite of what they appear to some American readers: not endorsements, but indictments, of the racist and anti-immigrant views of right-wing French politicians. After the murders, Rushdie tweeted that the president of SOS Racisme, the premier anti-racist group in France, had called Charlie Hebdo “the greatest anti-racist weekly in the country.” Justice Minister Christiane Taubira, whom opponents of the award described as “a black woman drawn as a monkey” in its pages, also paid tribute to the magazine.

I don’t agree that Charlie is racist—although its style of caricature apparently doesn’t travel well—and not just because Muslims are not a race. Charlie is against all forms of authoritarian religion (Le Monde analyzed 10 years of Charlie’s cover stories and found far more attacks on Christianity than on Islam). Indeed, it is blasphemous. Is that not an honorable left-wing thing to be? It used to be, before we became so confused about Islam: Half the time, we’re reminding one another that fundamentalists like the ones who committed the Charlie murders are a tiny fraction of the world’s 1.6 billion Muslims, who are ordinary people of goodwill, and the other half of the time we talk as if the murderers are out to redress real wrongs. Which is it? That latter view doesn’t serve Muslims well; it’s like saying people who assassinate abortion providers represent Christians, and West Bank settlers represent Jews.

Garry Trudeau accused Charlie of punching down—i.e., making jokes at the expense of the powerless. There’s something to that—why make struggling people unhappy?—but not enough. First, it assumes all French Muslims are offended. What about Mustapha Ourrad, the copy editor who was one of the murdered staffers? He was of Muslim descent, and so is Zineb El Rhazoui, who still writes for the magazine. Secondly, it’s saying you can make fun of Christianity, but Islam is out of bounds. That’s not how comedy works.

Charlie doesn’t mock Muslim people; it mocks fundamentalism—the narrow, bigoted, superstitious version of Islam that lies behind rather a lot of violence against writers. In the West, that violence began with the fatwa against Salman Rushdie for a handful of passages in The Satanic Verses that the Ayatollah Khomeini deemed grounds for death. Many of the writers targeted by fundamentalists have been Muslim themselves, among them the Bangladeshi writer and feminist Taslima Nasrin, the Saudi blogger Raif Badawi, and, going back a bit in time, the Nobel Prize–winning Egyptian novelist Naguib Mahfouz, thousands of Algerian intellectuals, and doubtless many more. These attacks had nothing to do with supposedly racist insults from privileged white people, and everything to do with perceived deviations from orthodoxy.

The six dissenters circulated a letter to PEN members, which as of this writing had over 200 signatures, including Joyce Carol Oates, Junot Diaz, Lorrie Moore, Janet Malcolm, and my Nation colleagues Jon Wiener and Dave Zirin. These writers must be awfully sure that they will never fall afoul of either fanaticism or well-meaning liberalism. “There is a critical difference between staunchly supporting expression that violates the acceptable,” the letter argues, “and enthusiastically rewarding such expression.” Well, sure, but excuse me: “violates the acceptable”? The acceptable what? And don’t we need writing and artwork that pushes the boundaries of what the acceptable is? “The road of excess leads to the palace of wisdom,” as Blake put it.

Norman Mailer, former president of PEN, had all sorts of reprehensible, ignorant views. When it came to politics, he was like a drunken uncle banging the table at Thanksgiving. But he pushed the boundaries back for every writer. And as much as I dislike a good deal of his writing and his repulsive ideas about women—talk about punching down—if PEN gave him an award, I would just live with it.

Stepping up its game, PEN announced six replacement table hosts: Art Spiegelman, Alison Bechdel, Neil Gaiman, Azar Nafisi, George Packer, and the French-Congolese novelist Alain Mabanckou, who will present the award to Charlie staffers Jean-Baptiste Thoret and Gérard Biard. The choice of Spiegelman was particularly brilliant: Here is a man who knows from offensive. When his Holocaust-survivor memoir Maus first came out, some called it tasteless (because if there’s one thing a book about mass genocidal sadism should be, it’s tasteful), and some Poles weren’t happy that he drew them as pigs. Just last month, Russian authorities banned Maus as “Nazi propaganda” because it has a swastika on the cover. They probably think Swift really proposed the Irish poor should eat their children.

Meanwhile, two armed men were killed after they opened fire, wounding a security guard at a Muhammad cartoon contest in Garland, Texas. The organizer of the show is Pamela Geller, a bigot previously known for putting up ads in subways calling Muslims savages. “The problem of Charlie Hebdo is absolutely not the same,” Thoret told Charlie Rose. “We fight racism,” added Biard. “And we have nothing to do with these people.”

Nonetheless, the Charlie cartoonist known as Luz says he will no longer draw Muhammad—not out of fear, but because “it no longer interests me.” Luz drew the famous post-massacre cover, in which Muhammad is shown holding a “Je suis Charlie” sign. The caption reads, “All is forgiven.”

Apparently not.