Eric Holder was the nation’s first black attorney general at a time when civil-rights issues were at the forefront of the public consciousness, from a new assault on voting rights to the Black Lives Matter movement to a rethinking of mass incarceration. He was often at the center of controversy around these issues, likening his tenure to Tupac Shakur’s “All Eyez on Me.”

Holder is an important character in my book Give Us the Ballot, and on Monday I had the opportunity to interview him at the Ford Foundation, as part of a fascinating symposium on “Reimagining Justice in the 21st Century” with Congressman John Lewis, Ford President Darren Walker, Heather McGee of Demos, Umi Selah of the Dream Defenders, Cristina Jimenez of United We Dream and Farhana Khera of Muslim Advocates. (My colleague Joan Walsh had a great piece about attending the gathering on the same day Donald Trump called for barring all Muslims from entering the United States.)

Holder denounced the gutting of the Voting Rights Act, praised the Black Lives Matter movement, said that American was still “a nation of cowards” when it came to discussing race, and took credit for rebuilding the Justice Department’s Civil Rights Division, which this week launched an investigation into the Chicago Police Department. He admitted that “the biggest failure I had as attorney general” was the Obama administration’s inability to persuade Congress to pass new gun-safety measures after Sandy Hook. And on a lighter note, he called Karl Rove “an idiot,” and said, “Biggie and Tupac were heard for the first time at the Justice Department when I was attorney general.”

Here are the highlights of our conversation.

Ari Berman : Do you think the election of the first African-American president and the appointment of the first African-American attorney general paradoxically led to a political climate where people believed that things like the Voting Rights Act were no longer needed?

Eric Holder: Yes. Probably some genuinely felt that. Some I think used the election of President Obama as an excuse to do away with those protections so hard won by people like Congressman Lewis. In the book you call these folks the “counter-revolutionaries” who were working to undermine the Voting Rights Act long before the election of Barack Obama. This gave them something to pin their arguments on, to say, “Look, why do we need a strong Voting Rights Act, we’ve got a black president.”

AB: You often noted in your speeches that there were more lawsuits challenging the constitutionality of Section 5 of the Voting Rights Act from 2011 from 2012 than during the previous four decades. Why? What changed?

EH: Some folks weren’t particularly happy with what an unfettered electorate did—they put a black man in the White House. At a whole bunch of levels, that was something certain people weren’t happy with and were determined to do something about.

For all that we thought President Obama’s election would mean—and it has meant substantial positive things—it has also galvanized the opposition. I think I was a little surprised by the intensity of the opposition and the way the opposition has operated.

AB: From 2011 to 2012, 180 new voting restrictions were introduced in 41 states and half the states in the country passed laws making it harder to vote. You were attorney general at the time. What was your reaction when this was happening and how did you shift the department’s priorities when you saw this?

EH: It became clear pretty quickly that attacking the right to vote, making it more difficult for the Obama coalition to be as effective as it had been in 2008, was going to be a focus of the opposition. So I made sure I got more resources for the Civil Rights Division generally and put more people in the Voting Section specifically to try to counter these measures.

AB: The Civil Rights Division was badly weakened when you took over. It went through a period of extreme politicization during the second Bush Administration. How did you try to rebuild it and how effectively is it functioning today?

EH: It has been restored. I hope it will be part of my legacy. I came in and had a Civil Rights Division that was decimated in numbers, that was low in morale and a sense of mission. The first visit I made as attorney general was to the Civil Rights Division to say, essentially, “this place is open for business again.”

AB: The case that struck down a key part of the Voting Rights Act was called Shelby County v. Holder. You are Holder. Do you think the Administration in retrospect could have done anything differently in arguing that case?

EH: No, I don’t think so. I was one of the few people in the department that thought there was no way the Supreme Court was going to go against the record Congress had established. Folks in the solicitor general’s office predicted the result.

With all due respect to the chief justice, this was the culmination of an effort he began many years before, when he was a young person in the Reagan Justice Department. He now had the ability to do what he was unable to do before. He had four votes in addition to his own and it really didn’t matter what we did, they were going to get to the result he desired.

AB: As a young lawyer in the Reagan Justice Department, John Roberts wrote memo after memo arguing that violations of the Voting Rights Act “should not be made too easy to prove.” He was writing not about Section 5 of the Voting Rights Act but about Section 2, which has been used by DOJ to challenge laws in Texas and North Carolina. How concerned are you that one of these cases is going to go before the Supreme Court and the Court is going to overturn the constitutionality of Section 2?

EH: Unlike what you hear in law school, the Supreme Court is a political institution and the reality is they can only go, I think, so far. Maybe I’m being exceedingly naïve, but I think that even for this Supreme Court, that’s a bridge too far. You gutted the act in the Shelby County decision and now to take out the remaining part, which we are now using to protect the right of people to vote, to knock that down or weaken that would subject the court to the kind of criticism that at least a couple of those justices would say, “We can’t go that far.”

AB: 2016 is the first presidential election in 50 years without the full protections of the Voting Rights Act. What should we be doing now to both protect voting rights in the short term but also in the longer term to reframe this discussion so that voter suppression is not the new normal?

EH: Well, I think in the short-term voter suppression is the new normal. We’ve got to be prepared to follow those laws that will be in place with regards to voter ID, with regards to the moving of polling places, we’ve got to be prepared for all of that. We’ve got to make sure that everyone who wants to cast a ballot can do that. That means having a familiarity with where you can vote, a familiarity with the kinds of things you need to be able to show to vote and then having an ability to know how to challenge the person or an institution that is making it harder for you to vote.

AB: There’s been a lot of talk about how Black Lives Matter has changed the public debate over civil rights. How has it changed people’s interpretation of the law? Toward the end of your tenure the Supreme Court heard a challenge to the Fair Housing Act and many people expected the Court to gut the Fair Housing Act. But they didn’t. And people thought that Justice Kennedy’s opinion may have been influenced by events in places like Baltimore and Ferguson.

I think that’s entirely possible. I did not expect that case to turn out the way that it did. The solicitor general sent me an e-mail or text that said, “We won,” and I thought, “Won what?”

That’s one of the values of Black Lives Matter. By being noisy, by being disruptive, Black Lives Matter is affecting the consciousness of this nation.

AB: One of the things it’s directly impacting is criminal-justice reform. The Washington Post said you were “known as a prosecutor who was unflinchingly tough on crime.” How have your own views on this issue evolved?

EH: I was a judge for five years and I left the bench after a while because I couldn’t stand the fact that I had an ocean of black men who came before me, who should have been the future of Washington, DC, that I was sending to jail for periods of time that I thought were inordinately long.

So when I became attorney general, especially after seeing what mass incarceration had done—the devastating impact it had on communities of color, the unfairness of it—it seemed to me that we had to reverse policies.

AB: Why are Republicans who are refusing to engage on the Voting Rights Act engaging on criminal-justice reform?

EH: They’re coming at it from different perspectives. Some say it’s unfair, it’s a question of justice. Others will look at it and say the way we run our government programs costs too much, it’s ineffective. One third of the Justice Department’s budget now goes to the bureau of prisons. From my perspective I don’t care what the reason is as long as we get to the same place.

AB: You famously said in 2009 that we were a “nation of cowards” when it came to discussing race. Do you think that’s still true?

EH: I still think that we as a nation are experts at not discussing things racial. When we’re forced to do it, when something happens in Ferguson, when something happens in Charleston, we deal with it for as short of a period of time as we can and then we retreat to our racial cocoons. As a result, lasting progress is not made, fundamental questions are not answered.

AB: We’re seeing a lot of anxiety about the changing demographics of the country. That’s one of the things that has led to a demonization of immigrants and refugees and Muslims and a new assault on voting rights. How do we protect rights and liberties for marginalized or previously disenfranchised communities in this age of racial anxiety?

EH: If we embrace these changes, the 21st century can be another American century. On the other hand, if we allow those on the margins and maybe not so on the margins, to use these demographic changes as a divisive force, we can really retard this progress—not only retard progress, we can go back. That’s why the vote is so important; at the end of the day, an unfettered electorate can have their say.